The compression fuzzers first try to invoke tj3LoadImage*() to load the
fuzz data, which allows for fuzzing the packed-pixel image loaders. If
tj3LoadImage*() fails, then the fuzzers try to construct a fake image
directly from the fuzz data. However, this requires at least two bytes
of data, since the first two bytes are used to derive the image
dimensions.
- Modify the existing fuzzers to extend default coverage to the
following code paths:
* ICC profile writing/reading, basic marker processing
* NULL color conversion with 3 or 4 components
* Huffman encoding with a destination buffer size < 512 bytes
* Partial image decompression
This was previously untouched by default because
2a5a3c6f0a was overly strict. Some
images will fail to decompress on the first iteration due to a
mismatch between the JPEG colorspace and the destination pixel
format. Thus, this commit modifies the decompression fuzzers to
detect whether the error thrown by tj3Decompress*() is due to an
excessive number of scans and bails out only in that case.
* Generating baseline-compatible quantization tables with low JPEG
quality levels
* 1/8 and 1/4 scaled IDCTs
- Add a new libjpeg-based decompression fuzzer that exercises code
paths not covered by the other fuzzers (or by the TurboJPEG API in
general):
* JCS_UNKNOWN (NULL color conversion with a component count other
than 3 or 4)
* Floating point IDCT
* Buffered-image mode
* Interstitial line skipping
* jpeg_save_markers() with a length limit
* Custom marker processor
Based on
5593bb138b
- Fall back to directly injecting fuzz data into the compressor if
tj3LoadImage*() fails. This prevents the packed-pixel image loaders
from acting as gatekeepers and thus preventing some fuzz data from
getting through to the codec.
Based on:
5593bb138b
- Add a JPEG dictionary to guide the decompression fuzzers.
Based on:
5593bb138b
- Remove duplication of effort in the cjpeg fuzzer.
Closes#845
- When built for fuzzing, expose special versions of tj3LoadImage*()
that accept a FILE handle.
- Rename cjpeg_main() to cjpeg_fuzzer() and allow a FILE handle to be
passed to it as well.
- Use fmemopen() to obtain a FILE handle from the fuzzer input data.
- Always use tj3Alloc() and tj3Free() for JPEG destination buffers.
- Always use malloc() and free() for other buffers.
Mixing and matching those functions works in the default implementation
of libjpeg-turbo on Un*x platforms, but it isn't guaranteed to work in
all implementations.
Closes#849
This is subtle, but #include <header.h> searches directories specified
with -I, then system include directories. #include "header.h" searches
the current source directory, then directories specified with -I, then
system include directories.
Using bracketed #include directives for jpeglib.h, jinclude.h, jerror.h,
cdjpeg.h, and turbojpeg.h only worked because the build system
explicitly passed -I{source_directory}/src/ to the compiler. Referring
to 51cee03629, it's better for the source
code to have as few dependencies on our specific build system as
possible.
Since jpeglib.h, jinclude.h, jerror.h, and turbojpeg.h can be installed
in system include directories, it's also better for internal references
to those headers to resolve internally first, to avoid potential
conflicts between the system-installed version of libjpeg-turbo and the
version being built. (Such conflicts would never have occurred with our
build system, but they might have occurred due to misintegration with a
downstream build system.)
Since the introduction of TJFLAG_NOREALLOC in libjpeg-turbo 1.2.x, the
TurboJPEG C API documentation has (confusingly) stated that:
- if the JPEG buffer pointer points to a pre-allocated buffer, then the
JPEG buffer size must be specified, and
- the JPEG buffer size should be specified if the JPEG buffer is
pre-allocated to an arbitrary size.
The documentation never explicitly stated that the JPEG buffer size
should be specified if the JPEG buffer is pre-allocated to a worst-case
size, but since focus does not imply exclusion, it also never explicitly
stated the reverse. Furthermore, the documentation never stated that
this was contingent upon TJPARAM_NOREALLOC/TJFLAG_NOREALLOC. However,
effectively the compression and lossless transformation functions
ignored the JPEG buffer size(s) passed to them, and assumed that the
JPEG buffer(s) had been allocated to a worst-case size, if
TJPARAM_NOREALLOC/TJFLAG_NOREALLOC was set. This behavior was an
accidental and undocumented throwback to libjpeg-turbo 1.1.x, in which
the tjCompress() function provided no way to specify the JPEG buffer
size. It was always a bad idea for applications to rely upon that
behavior (although our own TJBench application unfortunately did.)
However, if such applications exist in the wild, the new behavior would
constitute a breaking change, so it has been introduced only into
libjpeg-turbo 3.1.x and only into TurboJPEG 3 API functions. The
previous behavior has been retained when calling functions from the
TurboJPEG 2.1.x API and prior versions.
Did I mention that APIs are hard?
Referring to
https://sourceforge.net/p/libjpeg-turbo/bugs/48,
https://sourceforge.net/p/libjpeg-turbo/bugs/82,
#15, #238, #253, and #619,
valgrind and MSan have failed to properly detect data initialization by
libjpeg-turbo's x86 SIMD extensions for the entire 14 years that
libjpeg-turbo has been a project, resulting in false positives unless
libjpeg-turbo is built with WITH_SIMD=0 or run with JSIMD_FORCENONE=1.
This commit introduces a new C preprocessor macro (ZERO_BUFFERS) that,
if set, causes libjpeg-turbo to zero certain buffers in order to work
around the specific valgrind/MSan test failures caused by the
aforementioned false positives. This allows us to more closely
approximate the production configuration of libjpeg-turbo when testing
with valgrind or MSan.
Closes#781
This actually works and apparently always has worked. It only failed
because the libjpeg code, which did not originally support arithmetic
coding, assumed that optimize_coding should always be TRUE for 12-bit
data precision.
(ChangeLog update forthcoming)
- Prefix all function names with "tj3" and remove version suffixes from
function names. (Future API overhauls will increment the prefix to
"tj4", etc., thus retaining backward API/ABI compatibility without
versioning each individual function.)
- Replace stateless boolean flags (including TJ*FLAG_ARITHMETIC and
TJ*FLAG_LOSSLESS, which were never released) with stateful integer
parameters, the value of which persists between function calls.
* Use parameters for the JPEG quality and subsampling as well, in
order to eliminate the awkwardness of specifying function arguments
that weren't relevant for lossless compression.
* tj3DecompressHeader() now stores all relevant information about the
JPEG image, including the width, height, subsampling type, entropy
coding type, etc. in parameters rather than returning that
information in its arguments.
* TJ*FLAG_LIMITSCANS has been reimplemented as an integer parameter
(TJ*PARAM_SCANLIMIT) that allows the number of scans to be
specified.
- Use the const keyword for all pointer arguments to unmodified
buffers, as well as for both dimensions of 2D pointers. Addresses
#395.
- Use size_t rather than unsigned long to represent buffer sizes, since
unsigned long is a 32-bit type on Windows. Addresses #24.
- Return 0 from all buffer size functions if an error occurs, rather
than awkwardly trying to return -1 in an unsigned data type.
- Implement 12-bit and 16-bit data precision using dedicated
compression, decompression, and image I/O functions/methods.
* Suffix the names of all data-precision-specific functions with 8,
12, or 16.
* Because the YUV functions are intended to be used for video, they
are currently only implemented with 8-bit data precision, but they
can be expanded to 12-bit data precision in the future, if
necessary.
* Extend TJUnitTest and TJBench to test 12-bit and 16-bit data
precision, using a new -precision option.
* Add appropriate regression tests for all of the above to the 'test'
target.
* Extend tjbenchtest to test 12-bit and 16-bit data precision, and
add separate 'tjtest12' and 'tjtest16' targets.
* BufferedImage I/O in the Java API is currently limited to 8-bit
data precision, since the BufferedImage class does not
straightforwardly support higher data precisions.
* Extend the PPM reader to convert 12-bit and 16-bit PBMPLUS files
to grayscale or CMYK pixels, as it already does for 8-bit files.
- Properly accommodate lossless JPEG using dedicated parameters
(TJ*PARAM_LOSSLESS, TJ*PARAM_LOSSLESSPSV, and TJ*PARAM_LOSSLESSPT),
rather than using a flag and awkwardly repurposing the JPEG quality.
Update TJBench to properly reflect whether a JPEG image is lossless.
- Re-organize the TJBench usage screen.
- Update the Java docs using Java 11, to improve the formatting and
eliminate HTML frames.
- Use the accurate integer DCT algorithm by default for both
compression and decompression, since the "fast" algorithm is a legacy
feature, it does not pass the ISO compliance tests, and it is not
actually faster on modern x86 CPUs.
* Remove the -accuratedct option from TJBench and TJExample.
- Re-implement the 'tjtest' target using a CMake script that enables
the appropriate tests, depending on the data precision and whether or
not the Java API is part of the build.
- Consolidate the C and Java versions of tjbenchtest into one script.
- Consolidate the C and Java versions of tjexampletest into one script.
- Combine all initialization functions into a single function
(tj3Init()) that accepts an integer parameter specifying the
subsystems to initialize.
- Enable decompression scaling explicitly, using a new function/method
(tj3SetScalingFactor()/TJDecompressor.setScalingFactor()), rather
than implicitly using awkward "desired width"/"desired height"
parameters.
- Introduce a new macro/constant (TJUNSCALED/TJ.UNSCALED) that maps to
a scaling factor of 1/1.
- Implement partial image decompression, using a new function/method
(tj3SetCroppingRegion()/TJDecompressor.setCroppingRegion()) and
TJBench option (-crop). Extend tjbenchtest to test the new feature.
Addresses #1.
- Allow the JPEG colorspace to be specified explicitly when
compressing, using a new parameter (TJ*PARAM_COLORSPACE). This
allows JPEG images with the RGB and CMYK colorspaces to be created.
- Remove the error/difference image feature from TJBench. Identical
images to the ones that TJBench created can be generated using
ImageMagick with
'magick composite <original_image> <output_image> -compose difference <diff_image>'
- Handle JPEG images with unknown subsampling types. TJ*PARAM_SUBSAMP
is set to TJ*SAMP_UNKNOWN (== -1) for such images, but they can still
be decompressed fully into packed-pixel images or losslessly
transformed (with the exception of lossless cropping.) They cannot
be partially decompressed or decompressed into planar YUV images.
Note also that TJBench, due to its lack of support for imperfect
transforms, requires that the subsampling type be known when
rotating, flipping, or transversely transposing an image. Addresses
#436
- The Java version of TJBench now has identical functionality to the C
version. This was accomplished by (somewhat hackishly) calling the
TurboJPEG C image I/O functions through JNI and copying the pixels
between the C heap and the Java heap.
- Add parameters (TJ*PARAM_RESTARTROWS and TJ*PARAM_RESTARTBLOCKS) and
a TJBench option (-restart) to allow the restart marker interval to
be specified when compressing. Eliminate the undocumented TJ_RESTART
environment variable.
- Add a parameter (TJ*PARAM_OPTIMIZE), a transform option
(TJ*OPT_OPTIMIZE), and a TJBench option (-optimize) to allow
optimized baseline Huffman coding to be specified when compressing.
Eliminate the undocumented TJ_OPTIMIZE environment variable.
- Add parameters (TJ*PARAM_XDENSITY, TJ*PARAM_DENSITY, and
TJ*DENSITYUNITS) to allow the pixel density to be specified when
compressing or saving a Windows BMP image and to be queried when
decompressing or loading a Windows BMP image. Addresses #77.
- Refactor the fuzz targets to use the new API.
* Extend decompression coverage to 12-bit and 16-bit data precision.
* Replace the awkward cjpeg12 and cjpeg16 targets with proper
TurboJPEG-based compress12, compress12-lossless, and
compress16-lossless targets
- Fix innocuous UBSan warnings uncovered by the new fuzzers.
- Implement previous versions of the TurboJPEG API by wrapping the new
functions (tested by running the 2.1.x versions of TJBench, via
tjbenchtest, and TJUnitTest against the new implementation.)
* Remove all JNI functions for deprecated Java methods and implement
the deprecated methods using pure Java wrappers. It should be
understood that backward API compatibility in Java applies only to
the Java classes and that one cannot mix and match a JAR file from
one version of libjpeg-turbo with a JNI library from another
version.
- tj3Destroy() now silently accepts a NULL handle.
- tj3Alloc() and tj3Free() now return/accept void pointers, as malloc()
and free() do.
- The image I/O functions now accept a TurboJPEG instance handle, which
is used to transmit/receive parameters and to receive error
information.
Closes#517
