2/27/2018

Oodle 2.6.0 : Everything new and tasty

Oodle is an SDK for high performance lossless data compression. For more about Oodle, or licensing inquiries, visit the RAD Game Tools web site. This is my personal blog where I post supplemental material about Oodle.


A quick run down of all the exciting new stuff in Oodle 2.6.0 :

1. Leviathan!
2. The Kraken, Mermaid & Selkie fast-level encoders are now much faster.
3. Kraken & Mermaid's optimal level encoders now get more compression.
4. Kraken & Mermaid have new bit stream options which allow them to reach even higher compression.
5. Kraken and Mermaid are now more tuneable to different compression ratios and decode speeds.


1. Leviathan!

Read the Leviathan detailed performance report.


2. The Kraken, Mermaid & Selkie fast-level encoders are now much faster.

The non-optimal-parsing encoder levels that are intended for realtime use in Oodle are levels 1-4 aka SuperFast, VeryFast, Fast & Normal.

Their decode speed was always best in class, but previously their encode speed was slightly off the Pareto frontier (the best possible trade off of encode speed vs compression ratio). No longer.

"win81" test file (Core i7-3770 Windows x64)

Oodle 255 :

Kraken1   :  2.33:1 ,   83.3 enc MB/s ,  911.8 dec MB/s
Kraken2   :  2.39:1 ,   68.5 enc MB/s ,  938.0 dec MB/s
Kraken3   :  2.51:1 ,   24.4 enc MB/s , 1005.5 dec MB/s
Kraken4   :  2.57:1 ,   17.3 enc MB/s ,  997.3 dec MB/s

Oodle 260 :

Kraken1   :  2.33:1 ,  135.1 enc MB/s ,  928.1 dec MB/s
Kraken2   :  2.41:1 ,   94.0 enc MB/s ,  937.4 dec MB/s
Kraken3   :  2.52:1 ,   38.3 enc MB/s , 1020.6 dec MB/s
Kraken4   :  2.60:1 ,   23.0 enc MB/s , 1022.9 dec MB/s

Oodle 255 :

Mermaid1  :  2.10:1 ,  106.4 enc MB/s , 2079.4 dec MB/s
Mermaid2  :  2.15:1 ,   78.9 enc MB/s , 2161.1 dec MB/s
Mermaid3  :  2.24:1 ,   26.4 enc MB/s , 2294.0 dec MB/s
Mermaid4  :  2.29:1 ,   26.6 enc MB/s , 2341.1 dec MB/s

Oodle 260 :

Mermaid1  :  2.14:1 ,  161.1 enc MB/s , 2012.1 dec MB/s
Mermaid2  :  2.22:1 ,  104.1 enc MB/s , 2007.7 dec MB/s
Mermaid3  :  2.29:1 ,   39.0 enc MB/s , 2181.7 dec MB/s
Mermaid4  :  2.32:1 ,   32.1 enc MB/s , 2294.0 dec MB/s

Oodle 255 :

Selkie1   :  1.76:1 ,  146.8 enc MB/s , 3645.0 dec MB/s
Selkie2   :  1.85:1 ,  100.5 enc MB/s , 3565.0 dec MB/s
Selkie3   :  1.98:1 ,   28.2 enc MB/s , 3533.2 dec MB/s
Selkie4   :  2.04:1 ,   28.9 enc MB/s , 3675.9 dec MB/s

Oodle 260 :

Selkie1   :  1.78:1 ,  181.7 enc MB/s , 3716.0 dec MB/s
Selkie2   :  1.87:1 ,  114.7 enc MB/s , 3595.7 dec MB/s
Selkie3   :  1.98:1 ,   42.1 enc MB/s , 3653.1 dec MB/s
Selkie4   :  2.02:1 ,   34.9 enc MB/s , 3818.3 dec MB/s
The speed of the fastest encoder (level 1 = "SuperFast") is up by about 60% in Kraken & Mermaid.

Kraken's encode speed vs ratio is now competitive with ZStd, which has long been the best codec for encode speed tradeoff. For example, matching Kraken1 to the closest comparable ZStd levels on the same machine :

at similar encode speed, you can compare the compression ratios :

Kraken1       :  2.33:1 ,  135.1 enc MB/s ,  928.1 dec MB/s
zstd 1.3.3 -4 :  2.22:1 ,  136   enc MB/s ,  595   dec MB/s

or you can look at equal file sizes and compare the encode speed :

Kraken1       :  2.33:1 ,  135.1 enc MB/s ,  928.1 dec MB/s
zstd 1.3.3 -6 :  2:33:1 ,   62   enc MB/s ,  595   dec MB/s
Of course ZStd does have faster encode levels (1-3); Oodle does not provide anything in that domain.


3. Kraken & Mermaid's optimal level encoders now get more compression. (even with 2.5 compatible bit streams)

We improved the ability of the optimal parse encoders to make good decisions and find smaller encodings. This is at level 8 (Optimal4) our maximum compression level with slow encoding.

PD3D : (public domain 3D game data test set)

Kraken8 255    :  3.67:1 ,    2.8 enc MB/s , 1091.5 dec MB/s
Kraken8 260 -v5:  3.72:1 ,    1.2 enc MB/s , 1079.9 dec MB/s

GTS : (private game data test set)

Kraken8 255    :  2.60:1 ,    2.5 enc MB/s , 1335.8 dec MB/s
Kraken8 260 -v5:  2.63:1 ,    1.2 enc MB/s , 1343.8 dec MB/s

Silesia : (standard Silesia compression corpus)

Kraken8 255    :  4.12:1 ,    1.4 enc MB/s ,  982.0 dec MB/s
Kraken8 260 -v5:  4.18:1 ,    0.6 enc MB/s , 1018.7 dec MB/s

(speeds on Core i7-3770 Windows x64)
(-v5 means encode in v5 (2.5.x) backwards compatibility mode)
Compression ratio improvements around 1% might not sound like much, but when you're already on the Pareto frontier, finding another 1% without sacrificing any decode speed or changing the bit stream is quite significant.


4. Kraken & Mermaid have new bit stream options which allow them to reach even higher compression.

PD3D :

Kraken8 255    :  3.67:1 ,    2.8 enc MB/s , 1091.5 dec MB/s
Kraken8 260 -v5:  3.72:1 ,    1.2 enc MB/s , 1079.9 dec MB/s
Kraken8 260    :  4.00:1 ,    1.0 enc MB/s , 1034.7 dec MB/s

GTS :

Kraken8 255    :  2.60:1 ,    2.5 enc MB/s , 1335.8 dec MB/s
Kraken8 260 -v5:  2.63:1 ,    1.2 enc MB/s , 1343.8 dec MB/s
Kraken8 260    :  2.67:1 ,    1.0 enc MB/s , 1282.3 dec MB/s

Silesia :

Kraken8 255    :  4.12:1 ,    1.4 enc MB/s ,  982.0 dec MB/s
Kraken8 260 -v5:  4.18:1 ,    0.6 enc MB/s , 1018.7 dec MB/s
Kraken8 260    :  4.24:1 ,    0.6 enc MB/s ,  985.4 dec MB/s
Kraken in Oodle 2.6.0 now gets Silesia to 50,006,565 bytes at the default space-speed tradeoff target. Kraken in max-compression space-speed setting gets Silesia to 49,571,429 bytes (and is still far faster to decode than anything close).

If we look back at where Kraken started in April of 2016 , it was getting 4.05 to 1 on Silesia , now 4.24 to 1.

Kraken now usually gets more compression than anything remotely close to its decode speed.

Looking back at the old Performance of Oodle Kraken , Kraken only got 2.70:1 on win81. On some files, Kraken has always out-compressed the competition, but win81 was one where it lagged. It does better now :

"win81" test file (Core i7-3770 Windows x64)
in order of decreasing decode speed :

Kraken8       :  2.86:1 ,    0.6 enc MB/s ,  950.9 dec MB/s
old Kraken 215:  2.70:1 ,    1.0 enc mb/s ,  877.0 dec mb/s
Leviathan8    :  2.96:1 ,    0.4 enc MB/s ,  758.1 dec MB/s

zstd 1.3.3 -22           3.35 enc MB/s   473 dec MB/s    38804423  37.01 win81 = 2.702:1
zlib 1.2.11 -9           8.59 enc MB/s   254 dec MB/s    47516720  45.32 win81 = 2.206:1
brotli24 2017-12-12 -11  0.39 enc MB/s   214 dec MB/s    37246857  35.52 win81 = 2.815:1
lzham 1.0 -d26 -4        1.50 enc MB/s   158 dec MB/s    37794766  36.04 win81 = 2.775:1
lzma 16.04 -9            2.75 enc MB/s    51 dec MB/s    35890919  34.23 win81 = 2.921:1
At the time of Kraken's release, it was a huge decode speed win vs comparable compressors, but it sometimes lagged a bit in compression ratio. No longer.

NOTE : Oodle 2.6.0 by default makes bit streams that are decodable by version >= 2.6.0 only. If you need bit streams that can be read by earlier versions, you must set the backward compatible version number that you need. See the Oodle FAQ on backward compatibility.


5. Kraken and Mermaid are now more tuneable to different compression ratios and decode speeds.

The new v6 bit stream has more options, which allows them to smoothly trade off compression ratio for decode speed. The user can set this goal with a space-speed tradeoff parameter.

All the Oodle codecs have a compression level setting (similar to the familiar zip 1-9 level) that trades encode time for decode speed. Unlike many other codecs, Oodle's compressors do not lose *decode* speed at higher encode effort levels. We are not finding more compact encodings by making the decoder slower. Instead you can dial decode speed vs ratio with a separate parameter that changes how the encoder scores decisions.

More about this in : Oodle Hydra and space-speed flexibility.


Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility

or visit RAD to read for more information about the Oodle SDK

No comments:

old rants