How DxO's pioneering approach makes DNG files four times smaller without impacting quality

• Files four times smaller: linear DNG files from a 50 MP camera go from about 200 MB to just 50 MB. The linear DNG format thus becomes practical for everyday use, particularly for processing large series of images. Importing and synchronizing are faster, and these files obviously take up less disk space.
• High fidelity: the compression does not change the perception of the image, even with heavy edits.
• Compatibility: the resulting file remains a standard DNG file, which can be opened and edited with any application compatible with the DNG format (Adobe Lightroom, Capture One, etc.).

The linear DNG format is the output format that DxO recommends for DxO PureRAW, because it allows you to retain maximum editing flexibility without undermining the universal compatibility of images with third-party RAW processing software. However, even with the lossless compression built into the DNG specification, a linear DNG generally weighs about 4 MB per megapixel. For a 50 MP camera, that therefore represents 200 MB per image.

As you can see, it is therefore necessarily worthwhile to compress these files further.
But how far can you go without compromising quality?

Lossless compression is the most reassuring approach, both for developers and for users, since it guarantees that the decompressed file is mathematically identical to the original, bit for bit. This type of algorithm is, however, inherently limited in efficiency, particularly when the compressed signal contains information that is useless in the perceived rendering.

For DxO PureRAW 6, our imaging researchers developed a compression scheme that precisely targets this superfluous information: by removing it before compression, it is possible to achieve significantly higher compression ratios. The result is what is called visually lossless compression: the mathematical loss introduced is not perceptible to the human eye under normal viewing and editing conditions.

We identified two types of information that are irrelevant to the perceived rendering in linear DNG files:

1. Excessive pixel precision. The RAW files of digital cameras are generally encoded at 12 or 14 bits per pixel. At the output, our DeepPRIME processing chain uses 16 bits. However, the images always retain residual noise, which is deliberately maintained to avoid the artificial "plastic" look caused by total denoising. As we explain below, the more noise a signal contains, the less relevant it is to maintain total mathematical precision. Our dynamic range compression technology (Dynamic Range Compression, or DRC) is precisely designed to remove this useless precision.

2. The exact shape of textures and grain. In practice, the slight differences that exist between the exact shapes of the grain generated by noise or fine textures are imperceptible. Simplifying these micro-details is one of the great principles of image and video compression. That is precisely the role of the JPEG XL codec.

Both techniques rely on the standard mechanisms of the DNG format, so that all compatible software can open the resulting files without difficulty. DRC compression is encoded via the DNG linearization table tag, and the JPEG XL compression mode was introduced in DNG specification version 1.7. Both mechanisms are compatible with the main RAW processing applications.

Dynamic range compression (DRC) is a well-known technique in audio signal processing. A compressor reduces the dynamic range of a signal by applying a nonlinear transfer function: in the case of sound, loud parts are attenuated and quiet parts are amplified, so that the signal fits more efficiently into a given number of bits. This same principle adapts particularly well to digital RAW images.

Digital images are affected by photon noise (sometimes called "shot noise"), which is a fundamental property of light. The standard deviation of this noise grows with the square root of the signal intensity.
This has a major consequence for the compression of linear images:

• In dark areas, the noise is very low and the signal is finely structured. Each bit of precision can carry genuinely useful information: in this case, 14 or even 16 bits may prove necessary.
• In bright areas, the noise is higher. The useful precision of the signal is far below what can be represented with 14 or 16 bits. Thus, these extra bits encode the noise with a precision that no one needs and no one can perceive.

It is precisely these high-precision samples, useless because imperceptible in the highlights, that limit the efficiency of lossless compression: the compressor must precisely encode bits that carry no significant information.

• DRC compression solves the problem by applying a companding function (a curve close to the square root) to the linear pixel values before compression. On a theoretical level, this transformation is close to a variance-stabilizing transformation: after computing the square root, the standard deviation of the noise becomes approximately constant across the entire tonal range. Precision is thus targeted on what matters most, with a high number of levels in the shadows and far fewer in the highlights, but without ever removing visually perceptible information.

During decompression, the inverse function (stored in the DNG linearization table) restores the original linear encoding, exactly as provided for by the DNG specification. The process is completely transparent for downstream applications.

The number of quantization levels was determined by leaving a certain safety margin and tested in extreme editing scenarios (strong exposure increase combined with extreme recovery in the shadows) so that quantization artifacts remain invisible in all common uses.

After DRC compression, the processed image is compressed with JPEG XL, the next-generation image codec standardized by the JPEG committee.

The legacy JPEG format dates from 1992 and relies on a transformation in fixed 8 x 8 blocks that uses relatively simple entropy coding. Revolutionary for its time, this approach leaves considerable room for improvement in terms of compression performance by today's standards. The JPEG XL codec is the result of more than two decades of research into image compression:

Variable-size block transforms: the encoder can use large, efficient blocks in flat areas (up to 256 x 256) and very small, very precise blocks near edges (at minimum 2 x 2), thus adapting to the local content of the image instead of trying to impose a single size.

Perceptually optimized color space: the internal color representation of the JPEG XL format is modeled on the human visual system, which allows bits to be allocated more intelligently to the aspects of the image most important to perception.

Advanced entropy coding: modern coding techniques, significantly more efficient, can identify more redundancies in the data compared to traditional approaches.

Sophisticated prediction and contextual modeling: the encoder builds a statistical model of the image as processing progresses, to spot fine local structures and reduce the amount of truly unpredictable information to store.

Native handling of high bit depths: unlike the legacy JPEG format, the JPEG XL format was designed from the outset for high-bit-depth content, which makes it an ideal compression layer for RAW processing workflows.

We apply the JPEG XL codec with a near-lossless quality setting: the mathematical loss introduced by this codec is negligible, well below the noise floor of real images. Combining this compression with upstream DRC compression makes it extremely effective: by eliminating the precision of imperceptible details before passing the data to the JPEG XL codec, we provide it with a signal that is naturally easier to compress, without asking it to make decisions that could potentially be detrimental to quality.