TJ: Guard against reused JPEG dst buf w/0 buf size

The libjpeg in-memory destination manager has always re-allocated the
JPEG destination buffer if the specified buffer pointer is NULL or the
specified buffer size is 0.  TurboJPEG's destination manager inherited
that behavior.  Because of fe80ec2275,
TurboJPEG's destination manager tries to reuse the most recent
destination buffer if the same buffer pointer is specified.  (The
purpose of that is to enable repeated invocations of tj*Compress*() or
tj*Transform() to automatically grow the destination buffer, as needed,
with no intervention from the calling program.)  However, because of the
inherited code, TurboJPEG's destination manager also reallocated the
destination buffer if the specified buffer size was 0.  Thus, passing a
previously-used JPEG destination buffer pointer to tj*Compress*() or
tj*Transform() while specifying a destination buffer size of 0 confused
the destination manager.  It reallocated the destination buffer to 4096
bytes but reported the old destination buffer size to the libjpeg API.
This caused a buffer overrun if the old destination buffer size was
larger than 4096 bytes.

The documentation for tj*Compress*() is contradictory on this matter.
It states that the JPEG destination buffer size must be specified if the
destination buffer pointer is non-NULL.  However, it also states that,
if the destination buffer is reused, the specified destination buffer
size is ignored.  The documentation for tj*Transform() does not specify
the function's behavior if the destination buffer is reused.  Thus, the
behavior of the API is at best undefined if a calling application
attempts to reuse a destination buffer while specifying a destination
buffer size of 0.  If that ever worked, it only worked in libjpeg-turbo
1.3.x and prior.

This issue was exposed only through API abuse, and calling applications
that abused the API in that manner would not have worked for the last 11
years.  Thus, the issue did not represent a security threat.  This
commit merely hardens the API against such abuse, by modifying
TurboJPEG's destination manager so that it refuses to re-allocate the
JPEG destination buffer if the buffer pointer is reused and the
specified buffer size is 0.  That is consistent with the most permissive
interpretation of the TurboJPEG API documentation.  (The API already
ignored the destination buffer size if the destination buffer pointer
was reused and the specified buffer size was non-zero.  It makes sense
for it to do likewise if the specified buffer size is 0.)  This commit
also modifies TJUnitTest so that it verifies whether the API is hardened
against the aforementioned abuse.

ChangeLog.md

@@ -1,3 +1,13 @@
+3.1.3
+=====
+
+### Significant changes relative to 3.1.2:
+
+1. Hardened the TurboJPEG API against hypothetical applications that may
+erroneously call `tj*Compress*()` or `tj*Transform()` with a reused JPEG
+destination buffer pointer while specifying a destination buffer size of 0.
+
+
 3.1.2
 =====
 

src/jdatadst-tj.c

@@ -5,7 +5,7 @@
  * Copyright (C) 1994-1996, Thomas G. Lane.
  * Modified 2009-2012 by Guido Vollbeding.
  * libjpeg-turbo Modifications:
- * Copyright (C) 2011, 2014, 2016, 2019, 2022-2023, D. R. Commander.
+ * Copyright (C) 2011, 2014, 2016, 2019, 2022-2023, 2025, D. R. Commander.
  * For conditions of distribution and use, see the accompanying README.ijg
  * file.
  *
@@ -181,7 +181,7 @@ jpeg_mem_dest_tj(j_compress_ptr cinfo, unsigned char **outbuffer,
   dest->outsize = outsize;
   dest->alloc = alloc;
 
-  if (*outbuffer == NULL || *outsize == 0) {
+  if (*outbuffer == NULL || (*outsize == 0 && !reused)) {
     if (alloc) {
       /* Allocate initial buffer */
       dest->newbuffer = *outbuffer = (unsigned char *)MALLOC(OUTPUT_BUF_SIZE);

src/tjunittest.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (C)2009-2014, 2017-2019, 2022-2024 D. R. Commander.
+ * Copyright (C)2009-2014, 2017-2019, 2022-2025 D. R. Commander.
  *                                              All Rights Reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -769,6 +769,10 @@ static void bufSizeTest(void)
           TRY_TJ(handle, tj3EncodeYUV8(handle, (unsigned char *)srcBuf, w, 0,
                                        h, TJPF_BGRX, dstBuf, yuvAlign));
         } else {
+          /* Verify that the API is hardened against hypothetical applications
+             that may erroneously set the JPEG destination buffer size to 0
+             while reusing the destination buffer pointer. */
+          if (alloc && (w > 1 || h > 1)) dstSize = 0;
           if (precision <= 8) {
             TRY_TJ(handle, tj3Compress8(handle, (unsigned char *)srcBuf, w, 0,
                                         h, TJPF_BGRX, &dstBuf, &dstSize));
@@ -803,6 +807,7 @@ static void bufSizeTest(void)
           TRY_TJ(handle, tj3EncodeYUV8(handle, (unsigned char *)srcBuf, h, 0,
                                        w, TJPF_BGRX, dstBuf, yuvAlign));
         } else {
+          if (alloc && (w > 1 || h > 1)) dstSize = 0;
           if (precision <= 8) {
             TRY_TJ(handle, tj3Compress8(handle, (unsigned char *)srcBuf, h, 0,
                                         w, TJPF_BGRX, &dstBuf, &dstSize));