Several TurboJPEG functions store their return value in an unsigned
long long intermediate and compare it against the maximum value of
unsigned long or size_t in order to avoid integer overflow. However,
such comparisons are tautological (always true, i.e. redundant) unless
the size of unsigned long or size_t is less than the size of unsigned
long long. Explicitly guarding the comparisons with #if avoids compiler
warnings with -Wtautological-constant-in-range-compare in Clang and also
makes it clear to the reader that the comparisons are only intended for
32-bit code.
Refer to #752
(regression introduced by e8b40f3c2b)
The documented behavior of the libjpeg API is to compute optimal Huffman
tables when generating 12-bit lossy Huffman-coded JPEG images, unless
the calling application supplies its own Huffman tables. However,
e8b40f3c2b and
96bc40c1b3 modified
jinit_c_master_control() so that it always set cinfo->optimize_coding to
TRUE when generarating 12-bit lossy Huffman-coded JPEG images, which
prevented calling applications from supplying custom Huffman tables for
such images.
This commit modifies jinit_c_master_control() so that it only overrides
cinfo->optimize_coding when generating 12-bit lossy Huffman-coded JPEG
images if all Huffman table slots are empty or all slots contain default
Huffman tables. Determining whether the latter is true requires using
memcmp() to compare the allocated Huffman tables with the default
Huffman tables, because:
- The documented behavior of jpeg_set_defaults() is to initialize any
empty Huffman table slot with the default Huffman table corresponding
to that slot, regardless of the data precision. There is also no
requirement that the data precision be specified prior to calling
jpeg_set_defaults(). Thus, there is no reliable way to prevent
jpeg_set_defaults() from initializing empty Huffman table slots with
default Huffman tables, which are useless for 12-bit data precision.
- There is no requirement that custom Huffman tables be defined prior to
calling jpeg_set_defaults(). A calling application could call
jpeg_set_defaults() and modify the values in the default Huffman
tables rather than allocating new tables. Thus, there is no reliable
way to detect whether the allocated Huffman tables contain default
values without comparing the tables with the default Huffman tables.
Fortunately, comparing the allocated Huffman tables with the default
Huffman tables is the last stop on the logic train, so it won't happen
unless cinfo->data_precision == 12, cinfo->arith_code == FALSE,
cinfo->optimize_coding == FALSE, and one or more Huffman tables are
allocated. (If the compressor object is reused, this ensures that the
full comparison will be performed at most once.) Custom Huffman tables
will be flagged as non-default when the first non-default value is
encountered, and the worst case (comparing 400 bytes) is very fast on
modern CPUs anyhow.
Fixes#751
- Detect at configure time, via the __CET__ C preprocessor macro,
whether the C compiler will include either indirect branch tracking
(IBT) or shadow stack support, and define a NASM macro (__CET__) if
so.
- Modify the x86-64 SIMD code so that it includes appropriate endbr64
instructions (to support IBT) and an appropriate .note.gnu.property
section (to support both IBT and shadow stack) when __CET__ is
defined.
Closes#350
While QEMU will run executables from the current working directory,
other emulators may not. It is more reliable to pass the full
executable path to the emulator. The add_test(NAME ... COMMAND ...)
syntax automatically invokes the emulator (e.g. the command specified
in CMAKE_CROSSCOMPILING_EMULATOR) and passes the full executable path to
it, as long as the first COMMAND argument is the name of a target. This
cleans up the CMake code somewhat as well, since it is no longer
necessary to manually invoke CMAKE_CROSSCOMPILING_EMULATOR.
Closes#747
Disclaimer: I am not a lawyer, nor do I play one on TV.
Referring to #744, mentioning the zlib License as a license that applies
to libjpeg-turbo is confusing, and it isn't actually necessary, since
the IJG License subsumes the terms of the zlib License in the context of
the libjpeg API library and associated programs. This was presumably
understood to be the case by Miyasaka-san when he chose the zlib License
for the first libjpeg SIMD extensions. The libjpeg/SIMD web site
(https://cetus.sakura.ne.jp/softlab/jpeg-x86simd/jpegsimd.html) states
(translated from Japanese): "The terms of use of this SIMD enhanced
version of IJG JPEG software are subject to the terms of use of the
original version of IJG JPEG software."
Detailed analysis of the zlib License terms:
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any
damages arising from the use of this software.
This text is an almost literal subset of the warranty disclaimer text in
the IJG License. The IJG License states everything above with only
slight differences in wording, and it further clarifies that the user
assumes all risk as to the software's quality and accuracy and that
vendors of commercial products based on the software must assume all
warranty and liability claims.
Permission is granted to anyone to use this software for any
purpose, including commercial applications, and to alter it and
redistribute it freely, subject to the following restrictions:
This is semantically the same as the permission text in the IJG License,
since "use, copy, modify, and distribute this software (or portions
thereof) for any purpose, without fee" covers "use" for "any purpose,
including commercial applications" as well as alteration and
redistribution.
1. The origin of this software must not be misrepresented; you must
not claim that you wrote the original software. If you use this
software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
The IJG License requirement that "If any part of the source code for
this software is distributed, then this README file must be included,
with this copyright and no-warranty notice unaltered; and any additions,
deletions, or changes to the original files must be clearly indicated in
accompanying documentation" (Clause 1), as well as the requirement that
"If only executable code is distributed, then the accompanying
documentation must state that 'this software is based in part on the
work of the Independent JPEG Group'" (Clause 2), satisfies the
requirement of Clause 1 of the zlib License.
2. Altered source versions must be plainly marked as such, and must
not be misrepresented as being the original software.
Since Clause 1 of the IJG License applies only to the distribution of
source code, the copyright headers in the source code are effectively
"accompanying documentation" in that case. This is why we ensure that
the copyright headers of individual source files indicate the year(s) in
which modifications were made by each contributor. Doing so satisfies
the requirements of both Clause 2 of the zlib License and Clause 1 of
the IJG License.
3. This notice may not be removed or altered from any source
distribution.
Clauses 2 and 3 of the zlib License apply only to the source code that
bears that license. Thus, as applied to the software as a whole, those
requirements of the inbound zlib License are compatible with the
outbound IJG License as long as the IJG License does not contradict
them (which it doesn't.)
NOTE: To be clear, existing source code that bears the zlib License
cannot literally be re-licensed under the IJG License, since that would
violate Clause 3 of the zlib License. However, when considering the
terms under which the overall library is made available, the IJG License
effectively subsumes the terms of the zlib License.
https://www.gnu.org/licenses/license-compatibility.en.html is a
thorough, albeit somewhat GPL-biased, discussion of license
compatibility.
(regression introduced by 1644bdb7d2)
Setting a maximum version in cmake_minimum_required() effectively sets
the behavior to NEW for all policies introduced in all CMake versions up
to and including that maximum version. The NEW behavior for CMP0091,
introduced in CMake 3.15, uses CMake variables to specify the MSVC
runtime library against which to link, rather than placing the relevant
flags in CMAKE_C_FLAGS*. Thus, replacing /MD with /MT in CMAKE_C_FLAGS*
no longer has any effect when using CMake 3.15+.
- Move all libjpeg documentation, except for README.ijg, into the doc/
subdirectory.
- Move the TurboJPEG C API documentation from doc/html/ into
doc/turbojpeg/.
- Move all C source code and headers into a src/ subdirectory.
- Move turbojpeg-jni.c into the java/ subdirectory.
Referring to #226, there is no ideal solution to this problem. A
semantically ideal solution would have involved placing all source code,
including the SIMD and Java source code, under src/ (or perhaps placing
C library source code under lib/ and C test program source code under
test/), all header files under include/, and all documentation under
doc/. However:
- To me it makes more sense to have separate top-level directories for
each language, since the SIMD extensions and the Java API are
technically optional features. src/ now contains only the code that
is relevant to the core C API libraries and associated programs.
- I didn't want to bury the java/ and simd/ directories or add a level
of depth to them, since both directories already contain source code
that is 3-4 levels deep.
- I would prefer not to separate the header files from the C source
code, because:
1. It would be disruptive. libjpeg and libjpeg-turbo have
historically placed C source code and headers in the same
directory, and people who are familiar with both projects (self
included) are used to looking for the headers in the same directory
as the C source code.
2. In terms of how the headers are used internally in libjpeg-turbo,
the distinction between public and private headers is a bit fuzzy.
- It didn't make sense to separate the test source code from the library
source code, since there is not a clear distinction in some cases.
(For instance, the IJG image I/O functions are used by cjpeg and djpeg
as well as by the TurboJPEG API.)
This solution is minimally disruptive, since it keeps all C source code
and headers together and keeps java/ and simd/ as top-level directories.
It is a bit awkward, because java/ and simd/ technically contain source
code, even though they are not under src/. However, other solutions
would have been more awkward for different reasons.
Closes#226
- Integrate
c07bba2730 (diff-1e2deb5301e9481203102fcddd1b2d0d2bf0ddc1cbb445c7f4b6414a3b869ce8)
so that the default man directory is <CMAKE_INSTALL_PREFIX>/share/man
on FreeBSD systems.
- For good measure, integrate
f835f189ae
so that the default info directory is
<CMAKE_INSTALL_PREFIX>/share/info on FreeBSD systems, even though we
don't use that directory.
- Automatically set the CMake variable type to PATH for any
user-specified CMAKE_INSTALL_*DIR variables.
Addresses concerns raised in #326, #346, #648Closes#648
libjpeg-turbo always includes Adobe APP14 markers in the lossless JPEG
images that it generates, but some compressors (e.g. accusoft PICTools
Medical) do not.
Fixes#743
ef9a4e05ba (libjpeg-turbo 1.4.x), which
was based on
https://bug815473.bmoattachments.org/attachment.cgi?id=692126
(https://bugzilla.mozilla.org/show_bug.cgi?id=815473), modified the C
baseline Huffman encoder so that it precomputes jpeg_nbits_table, in
order to facilitate sharing the table among multiple processes.
However, libjpeg-turbo never shared the table, and because the table was
implemented as a static array, f3a8684cd1
(libjpeg-turbo 1.5.x) and 37bae1a0e9
(libjpeg-turbo 2.0.x) each introduced a duplicate copy of the table for
(respectively) the SSE2 baseline Huffman encoder and the C progressive
Huffman encoder.
This commit does the following:
- Move the duplicated code in jchuff.c and jcphuff.c, originally
introduced in 0cfc4c17b7 and
37bae1a0e9, into a header
(jpeg_nbits.h).
- Credit the co-author of 0cfc4c17b7.
(Refer to https://sourceforge.net/p/libjpeg-turbo/patches/57).
- Modify the SSE2 baseline Huffman encoder so that the C Huffman
encoders can share its definition of jpeg_nbits_table.
- Move the definition of jpeg_nbits_table into a C source file
(jpeg_nbits.c) rather than a header, and define the table only if
USE_CLZ_INTRINSIC is undefined and the SSE2 baseline Huffman encoder
will not be built.
- Apply hidden symbol visibility to the shared definition of
jpeg_nbits_table, if the compiler supports the necessary attribute.
(In practice, only Visual C++ doesn't.)
Closes#114
See also:
https://bugzilla.mozilla.org/show_bug.cgi?id=1501523
- Clarify that lossless JPEG is slower than and doesn't compress as well
as lossy JPEG. (That should be obvious, because "lossy" literally
means that data is thrown away.)
- Re-generate TurboJPEG C API documentation using Doxygen 1.9.8.
- Clarify that setting the data_precision field in jpeg_compress_struct
to 16 requires lossless mode.
- Explain what the predictor selection value actually does. (Refer to
Recommendation ITU-T T.81 (1992) | ISO/IEC 10918-1:1994, Section
H.1.2.1.)
TJPARAM_MAXPIXELS was previously hidden and used only for fuzz testing,
but it is potentially useful for calling applications as well,
particularly if they want to guard against excessive memory consumption
by the tj3LoadImage*() functions. The parameter has also been extended
to decompression and lossless transformation functions/methods, mainly
as a convenience. (It was already possible for calling applications to
impose their own JPEG image size limits by reading the JPEG header prior
to decompressing or transforming the image.)
Unfortunately, most of the GitHub security advisories filed against
libjpeg-turbo thus far have been the result of non-exploitable API
abuses triggered by randomly-generated test programs and accompanied by
wild claims of denials of service with no demonstrable or even probable
exploit that might cause such a DoS (assuming a service even existed
that used the API in question.) Security advisories remain private
unless accepted, and I cannot accept them if they do not describe an
actual security issue. Thus, it's best to steer most users toward
regular bug reports.
Because of the TurboJPEG 3 API overhaul, the legacy decompression and
lossless transformation functions now wrap the new TurboJPEG 3
functions. For performance reasons, we don't want to read the JPEG
header more than once during the same operation, so the wrapped
functions do not read the header if it has already been read by a
wrapper function. Initially the TurboJPEG 3 functions used a state
variable to track whether the header had already been read, but
b94041390c made this more robust by using
the libjpeg global decompression state instead. If a wrapper function
has already read the JPEG header successfully, then the global
decompression state will be DSTATE_READY, and the logic introduced in
b94041390c will prevent the header from
being read again.
A subtle issue arises because tj3DecompressHeader() does not call
jpeg_abort_decompress() if jpeg_read_header() fails. (That is arguably
a bug, but it has existed since the very first implementation of the
function.) Depending on the nature of the failure, this can cause
tj3DecompressHeader() to return an error code and leave the libjpeg
global decompression state set to DSTATE_INHEADER. If a misbehaved
application ignored the error and subsequently called a TurboJPEG
decompression or lossless transformation function, then the function
would fail to read the JPEG header because the global decompression
state was greater than DSTATE_START. In the case of the decompression
functions, this was innocuous, because jpeg_calc_output_dimensions()
and jpeg_start_decompress() both sanity check the global decompression
state. However, it was possible for a misbehaved application to call
tj3DecompressHeader() with junk data, ignore the return value, and pass
the same junk data into tj3Transform(). Because tj3DecompressHeader()
left the global decompression state set to DSTATE_INHEADER,
tj3Transform() failed to detect the junk data (because it didn't try to
read the JPEG header), and it called jtransform_request_workspace() with
dinfo->image_width and dinfo->image_height still initialized to 0.
Because jtransform_request_workspace() does not sanity check the
decompression state, a division-by-zero error occurred with certain
combinations of transform options in which TJXOPT_TRIM or TJXOPT_CROP
was specified. However, it should be noted that TJXOPT_TRIM and
TJXOPT_CROP cannot be expected to work properly without foreknowledge of
the JPEG source image dimensions, which cannot be gained except by
calling tj3DecompressHeader() successfully. Thus, a calling application
is inviting trouble if it does not check the return value of
tj3DecompressHeader() and sanity check the JPEG source image dimensions
before calling tj3Transform(). This commit softens the failure, but the
failure is still due to improper API usage.
(regression introduced by 300a344d65)
300a344d65 fixed the recompression code
path but also broke the pure decompression code path, because the fix
caused the TurboJPEG decompression instance to be destroyed before
tj3SaveImage() could use it. Furthermore, the fix in
300a344d65 prevented pixel density
information from being transferred from the input image to the output
image when recompressing. This commit does the following:
- Modify tjexample.c so that a single TurboJPEG instance is initialized
for lossless transformation and shared by all code paths. In addition
to fixing both of the aforementioned issues, this makes the code more
readable.
- Extend tjexampletest to test the recompression code path, thus
ensuring that the issues fixed by this commit and
300a344d65 are not reintroduced.
- Modify tjexample.c to remove redundant fclose(), tj3Destroy(), and
tj3Free() calls.
This corresponds to max_memory_to_use in the jpeg_memory_mgr struct in
the libjpeg API, except that the TurboJPEG parameter is specified in
megabytes. Because this is 2023 and computers with less than 1 MB of
memory are not a thing (at least not within the scope of libjpeg-turbo
support), it isn't useful to allow a limit less than 1 MB to be
specified. Furthermore, because TurboJPEG parameters are signed
integers, if we allowed the memory limit to be specified in bytes, then
it would be impossible to specify a limit larger than 2 GB on 64-bit
machines. Because max_memory_to_use is a long signed integer,
effectively we can specify a limit of up to 2 petabytes on 64-bit
machines if the TurboJPEG parameter is specified in megabytes. (2 PB
should be enough for anybody, right?)
This commit also bumps the TurboJPEG API version to 3.0.1. Since the
TurboJPEG API version no longer tracks the libjpeg-turbo version, it
makes sense to increment the API revision number when adding constants,
to increment the minor version number when adding functions, and to
increment the major version number for a complete overhaul.
This commit also removes the vestigial TJ_NUMPARAM macro, which was
never defined because it proved unnecessary.
Partially implements #735
This is very subtle, but if a user specifies a libjpeg virtual array
memory limit via the JPEGMEM environment variable and one of the
tj3Compress*() functions hits that limit, the libjpeg error handler
will be invoked in jpeg_start_compress() (more specifically in
realize_virt_arrays() in jinit_compress_master()) before the libjpeg
global compression state can be incremented. Thus,
jpeg_abort_compress() will not be called before the tj3Compress*()
function exits, the unrealized virtual arrays will not be freed, and if
the TurboJPEG compression instance is reused, those unrealized virtual
arrays will count against the specified memory limit. This could cause
subsequent compression operations that require smaller virtual arrays
(or even no virtual arrays at all) to fail when they would otherwise
succeed. In reality, the vast majority of calling programs would abort
and free the TurboJPEG compression instance if one of the tj3Compress*()
functions failed, but TJBench is a rare exception. This issue does not
bear documenting because of its subtlety and rarity and because JPEGMEM
is not a documented feature of the TurboJPEG API.
Note that the issue does not exist in the tj3Encode*() and tj3Decode*()
functions, because realize_virt_arrays() is never called in the body of
those functions. The issue also does not exist in the tj3Decompress*()
and tj3Transform() functions, because those functions ensure that the
JPEG header is read (and thus the libjpeg global decompression state is
incremented) prior to calling a function that calls
realize_virt_arrays() (i.e. jpeg_start_decompress() or
jpeg_read_coefficients().) If realize_virt_arrays() failed in the body
of jpeg_write_coefficients(), then tj3Transform() would abort without
calling jpeg_abort_compress(). However, since jpeg_start_compress() is
never called in the body of tj3Transform(), no virtual arrays are ever
requested from the compression object, so failing to call
jpeg_abort_compress() would be innocuous.
Those obsolete memory managers have never been included in
libjpeg-turbo, nor has libjpeg-turbo ever claimed to support MS-DOS
or Mac operating systems prior to OS X 10.4 "Tiger." Note that we
retain the ability to use temp files, even though libjpeg-turbo does
not use them. This allows downstream implementations to write their own
memory managers that use temp files, if necessary.
If the align parameter was set to an unreasonably large value, such as
0x2000000, strides[0] * ph0 and strides[1] * ph1 could have overflowed
the int datatype and wrapped around when computing (src|dst)Planes[1]
and (src|dst)Planes[2] (respectively.) This would have caused
(src|dst)Planes[1] and (src|dst)Planes[2] to point to lower addresses in
the YUV buffer than expected, so the worst case would have been a
visually incorrect output image, not a buffer overrun or other
exploitable issue.
Because of 47656a0820, we can now
reliably determine the correct default values for FLOATTEST8 and
FLOATTEST12 when using Clang or GCC to build for AArch64 or PowerPC
platforms. (Testing confirms that this is the case with GCC 5-13 and
Clang 5-14 on Ubuntu/AArch64, GCC 4 on CentOS 7/PPC, and GCC 8-10 and
Clang 6-12 on Ubuntu/PPCLE.) Other CPU architectures and compilers can
be added on a case-by-case basis as they are tested.
Because of bf01ed2fbc, the simd field in
huff_entropy_encoder (and, by extension, the simd field in
savable_state) is only initialized if WITH_SIMD is defined. Due to an
oversight, the simd field in savable_state was queried in flush_bits()
regardless of whether WITH_SIMD was defined. In most cases, both
branches of the query have identical code, and the optimizer removes the
branch. However, because the legacy Neon GAS Huffman encoder uses the
older bit buffer logic from libjpeg-turbo 2.0.x and prior (refer to
087c29e07f), the branches do not have
identical code when building for AArch64 with NEON_INTRINSICS undefined
(which will be the case if WITH_SIMD is undefined.) Thus, if
libjpeg-turbo was built for AArch64 with the SIMD extensions disabled
at build time, it was possible for the Neon GAS branch in flush_bits()
to be taken, which would have set put_bits to a value that is incorrect
for the C Huffman encoder. Referring to #728, a user reported that this
issue sometimes caused libjpeg-turbo to generate bogus JPEG images if it
was built for AArch64 without SIMD extensions and subsequently used
through the Qt framework. (It should be noted, however, that disabling
the SIMD extensions in AArch64 builds of libjpeg-turbo is inadvisable
for performance reasons.)
I was unable to reproduce the issue on Linux/AArch64 using libjpeg-turbo
alone, despite testing various versions of GCC and Clang and various
optimization levels. However, the issue is reproducible using MSan with
-O0, so this commit also modifies the GitHub Actions workflow so that
compiler optimization is disabled in the linux-msan job. That should
prevent the issue or similar issues from re-emerging.
Fixes#728
libjpeg-turbo implements 4:4:0 "fancy" (smooth) upsampling, which is
enabled by default when decompressing JPEG images that use 4:4:0
chrominance subsampling. libjpeg did not and does not implement fancy
4:4:0 upsampling.
