0
0
mirror of https://github.com/lz4/lz4 synced 2026-01-18 17:21:30 +01:00

Merge pull request #1643 from Benjamin-Philip/bp/docs-touch-up-double-buffer-example

Touch up double buffer example
This commit is contained in:
Yann Collet
2026-01-01 22:48:27 -08:00
committed by GitHub

View File

@@ -1,26 +1,31 @@
# LZ4 Streaming API Example : Double Buffer
# LZ4 Streaming API Example: Double Buffer
by *Takayuki Matsuoka*
`blockStreaming_doubleBuffer.c` is LZ4 Streaming API example which implements double buffer (de)compression.
[`blockStreaming_doubleBuffer.c`](blockStreaming_doubleBuffer.c) is an example
of the LZ4 Streaming API where we implement double buffer (de)compression.
Please note :
Please note:
- Firstly, read "LZ4 Streaming API Basics".
- This is relatively advanced application example.
- Output file is not compatible with lz4frame and platform dependent.
- Firstly, read ["LZ4 Streaming API Basics"](streaming_api_basics.md).
- This is a relatively advanced application example.
- The output file is not compatible with lz4frame and is platform dependent.
## What's the point of this example ?
## What's the point of this example?
- Handle huge file in small amount of memory
- Always better compression ratio than Block API
- Uniform block size
The LZ4 Streaming API can be used to handle compressing a huge file in a small
amount of memory. This example shows how to use a "Double Buffer" to compress
blocks (i.e. chunks) of a uniform size in a stream. The Streaming API used in
this way *always* yields a better compression ratio than the regular Block API.
For an example with non-uniform blocks, see ["LZ4 Streaming API Example: Line by
Line Text Compression"](blockStreaming_lineByLine.md).
## How the compression works
## How compression works
First of all, allocate "Double Buffer" for input and LZ4 compressed data buffer for output.
Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
Firstly, allocate "Double Buffer" for input and "compressed data buffer" for
output. Double buffer has two pages, the "first" page (`Page#1`) and the "second"
page (`Page#2`).
```
Double Buffer
@@ -32,10 +37,17 @@ Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
|
v
{Out#1}
```
Next, read the first block to the double buffer's first page. Compress it with
`LZ4_compress_continue()`. On the first compression, LZ4 doesn't have any
previous dependencies, so it just compresses the block without dependencies and
writes the compressed block `{Out#1}` to the compressed data buffer. After that,
write `{Out#1}` to the file.
```
Prefix Dependency
+---------+
| |
v |
+---------+----+----+
@@ -44,8 +56,13 @@ Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
|
v
{Out#2}
```
Next, read the second block to the double buffer's second page and compress it.
This time, LZ4 can use the dependency on `Block#1` to improve the compression
ratio. This dependency is called "Prefix mode".
```
External Dictionary Mode
+---------+
| |
@@ -56,8 +73,13 @@ Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
|
v
{Out#3}
```
Next, read the third block to the double buffer's *first* page and compress it.
This time LZ4 can use dependency on Block#2. This dependency is called "External
Dictionary mode".
```
Prefix Dependency
+---------+
| |
@@ -70,31 +92,18 @@ Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
{Out#4}
```
Next, read first block to double buffer's first page. And compress it by `LZ4_compress_continue()`.
For the first time, LZ4 doesn't know any previous dependencies,
so it just compress the line without dependencies and generates compressed block {Out#1} to LZ4 compressed data buffer.
After that, write {Out#1} to the file.
Next, read second block to double buffer's second page. And compress it.
This time, LZ4 can use dependency to Block#1 to improve compression ratio.
This dependency is called "Prefix mode".
Next, read third block to double buffer's *first* page, and compress it.
Also this time, LZ4 can use dependency to Block#2.
This dependency is called "External Dictonaly mode".
Continue these procedure to the end of the file.
Continue this procedure till the end of the file.
## How the decompression works
## How decompression works
Decompression will do reverse order.
Decompression follows the reverse order:
- Read first compressed block.
- Decompress it to the first page and write that page to the file.
- Read second compressed block.
- Decompress it to the second page and write that page to the file.
- Read third compressed block.
- Decompress it to the *first* page and write that page to the file.
- Read the first compressed block.
- Decompress it to the first page and write that page to the file.
- Read the second compressed block.
- Decompress it to the second page and write that page to the file.
- Read the third compressed block.
- Decompress it to the *first* page and write that page to the file.
Continue these procedure to the end of the compressed file.
Continue this procedure till the end of the compressed file.