You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by "JakeChampion (via GitHub)" <gi...@apache.org> on 2025/04/18 08:08:39 UTC

[PR] Add Zstandard compression support and update tests [trafficserver]

JakeChampion opened a new pull request, #12201:
URL: https://github.com/apache/trafficserver/pull/12201

   - Implemented Zstandard (zstd) compression in the compress plugin.
   - Updated CMake configuration to include zstd.
   - Enhanced the normalization of the Accept-Encoding header to support zstd.
   - Added tests for zstd compression functionality.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2285118510


##########
plugins/compress/compress.cc:
##########
@@ -136,8 +132,15 @@ handle_range_request(TSMBuffer req_buf, TSMLoc req_loc, HostConfiguration *hc)
 }
 } // namespace
 
+// Forward declarations for ZSTD compression functions
+#if HAVE_ZSTD_H
+static void zstd_compress_init(Data *data);
+static void zstd_compress_finish(Data *data);
+static void zstd_compress_one(Data *data, const char *upstream_buffer, int64_t upstream_length);

Review Comment:
   Thank you, done in [`03e647a` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/03e647a7ebb7c5dbcb6552fe11cd34a527a75e4f)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2281757566


##########
cmake/Findzstd.cmake:
##########
@@ -0,0 +1,53 @@
+#######################

Review Comment:
   good point, implemented via [`17e3c6b` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/17e3c6b49373f5b35977f420719f18d0ec3b3efb)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2283670572


##########
src/traffic_layout/info.cc:
##########
@@ -91,6 +95,11 @@ produce_features(bool json)
 #else
   print_feature("TS_HAS_BROTLI", 0, json);
 #endif
+#if HAVE_ZSTD_H
+  print_feature("TS_HAS_ZSTD", 1, json);
+#else
+  print_feature("TS_HAS_ZSTD", 0, json);
+#endif

Review Comment:
   ```suggestion
   #ifdef HAVE_ZSTD_H
     print_feature("TS_HAS_ZSTD", 1, json);
   #else
     print_feature("TS_HAS_ZSTD", 0, json);
   #endif
   ```



##########
src/traffic_layout/info.cc:
##########
@@ -203,6 +212,11 @@ produce_versions(bool json)
 #else
   print_var("brotli", undef, json);
 #endif
+#if HAVE_ZSTD_H
+  print_var("zstd", LBW().print("{}", ZSTD_versionString()).view(), json);
+#else
+  print_var("zstd", undef, json);
+#endif

Review Comment:
   ```suggestion
   #ifdef HAVE_ZSTD_H
     print_var("zstd", LBW().print("{}", ZSTD_versionString()).view(), json);
   #else
     print_var("zstd", undef, json);
   #endif
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-2821006598

   Looks like all the FreeBSD machines for CI are offline


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3453451382

   I think the code is fine at this point, but I have a few concerns:
   
    - I'm not convinced autests will pass if the library is not available.
    - There is some code (transact headers, xpack) that unconditionally expect zstd
    - Maybe we should just require zstd for the build (probably not, and would have to be done for ATS 11)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3575288093

   @bryancall I forgot to tag you, apologies - I fixed all the conflicts and the tests pass once again now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bneradt (via GitHub)" <gi...@apache.org>.
bneradt commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2427484008


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   Probably don't want this in the docs. :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2486648689


##########
contrib/docker/ubuntu/noble/Dockerfile:
##########
@@ -48,6 +48,9 @@ RUN apt update \
     libpcre3-dev \
     hwloc \
     libbrotli-dev \
+    libzstd-dev \
+    luajit \
+    luajit \

Review Comment:
   done in [`88a02b2` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/88a02b2fdd4f8ba6d7cbe50111d0996aae064b7c)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3071253518

   [approve ci autest]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2535442183


##########
plugins/compress/compress.cc:
##########
@@ -580,7 +620,14 @@ transformable(TSHttpTxn txnp, bool server, HostConfiguration *host_configuration
         continue;
       }
 
-      if (strncasecmp(value, "br", sizeof("br") - 1) == 0) {
+      info("Accept-Encoding value [%.*s]", len, value);

Review Comment:
   ```suggestion
         debug("Accept-Encoding value [%.*s]", len, value);
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "maskit (via GitHub)" <gi...@apache.org>.
maskit commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2430406718


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   Trimming the whitespace would be nice, but I'd like to see the change on a separate PR. One change on one PR.
   
   @bneradt I'm really not sure what makes you think this is not very user-friendly. It looks like an important note for users with a configuration example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "masaori335 (via GitHub)" <gi...@apache.org>.
masaori335 commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-2816346401

   [approve ci]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "zwoop (via GitHub)" <gi...@apache.org>.
zwoop commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2512007497


##########
plugins/compress/compress.cc:
##########
@@ -580,7 +620,14 @@ transformable(TSHttpTxn txnp, bool server, HostConfiguration *host_configuration
         continue;
       }
 
-      if (strncasecmp(value, "br", sizeof("br") - 1) == 0) {
+      info("Accept-Encoding value [%.*s]", len, value);

Review Comment:
   Nit picking (and I know it's the same), but this feels more like a debug() than an info().  I'm honestly not sure why this plugin has debug / info / warning all going to the same Dbg() tag.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2430181443


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   It is yeah, we could opt to trim the whitespace so that it doesn't matter if they included it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3543737786

   > @JakeChampion looks like there is now a conflict and should look at the comment from @zwoop above about info vs debug
   
   I've fixed conflicts and changed the call from info to debug now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2428183357


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   which part should we remove, is it the original sentence noting whitespace?
   
   ```suggestion
   ``supported-algorithms deflate,gzip,br,zstd``.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3474740205

   ## Code Review - ZSTD Compression Implementation
   
   Thank you for this excellent work on adding ZSTD compression support! The testing shows the feature works well and achieves great compression ratios (98.6%). I've completed a thorough code review and found a few issues that should be addressed before merging.
   
   ---
   
   ## 🔴 Critical Issue
   
   ### Resource Management in `zstd_transform_init()`
   
   **Location**: `plugins/compress/compress.cc:558-580`
   
   **Problem**: If `ZSTD_CCtx_setParameter` fails during initialization, the function returns early but leaves `data->zstrm_zstd.cctx` in a valid (non-NULL) state. The caller at line 395-398 only checks `if (!data->zstrm_zstd.cctx)`, which won't catch these initialization failures. This could lead to compression proceeding with a misconfigured context.
   
   **Current Code**:
   ```cpp
   static void
   zstd_transform_init(Data *data)
   {
     if (!data->zstrm_zstd.cctx) {
       error("Failed to initialize Zstd compression context");
       return;
     }
   
     size_t result = ZSTD_CCtx_setParameter(data->zstrm_zstd.cctx, ZSTD_c_compressionLevel, ...);
     if (ZSTD_isError(result)) {
       error("Failed to set Zstd compression level: %s", ZSTD_getErrorName(result));
       return;  // âš ī¸ Context is valid but misconfigured
     }
     
     result = ZSTD_CCtx_setParameter(data->zstrm_zstd.cctx, ZSTD_c_checksumFlag, 1);
     if (ZSTD_isError(result)) {
       error("Failed to enable Zstd checksum: %s", ZSTD_getErrorName(result));
       return;  // âš ī¸ Context exists but checksum not enabled
     }
   }
   ```
   
   **Recommended Fix**:
   ```cpp
   static void
   zstd_transform_init(Data *data)
   {
     if (!data->zstrm_zstd.cctx) {
       error("Failed to initialize Zstd compression context");
       return;
     }
   
     size_t result = ZSTD_CCtx_setParameter(data->zstrm_zstd.cctx, ZSTD_c_compressionLevel, data->hc->zstd_compression_level());
     if (ZSTD_isError(result)) {
       error("Failed to set Zstd compression level: %s", ZSTD_getErrorName(result));
       ZSTD_freeCCtx(data->zstrm_zstd.cctx);  // ✅ Clean up
       data->zstrm_zstd.cctx = nullptr;        // ✅ Mark as invalid
       return;
     }
     
     result = ZSTD_CCtx_setParameter(data->zstrm_zstd.cctx, ZSTD_c_checksumFlag, 1);
     if (ZSTD_isError(result)) {
       error("Failed to enable Zstd checksum: %s", ZSTD_getErrorName(result));
       ZSTD_freeCCtx(data->zstrm_zstd.cctx);  // ✅ Clean up
       data->zstrm_zstd.cctx = nullptr;        // ✅ Mark as invalid
       return;
     }
   
     debug("zstd compression context initialized with level %d", data->hc->zstd_compression_level());
   }
   ```
   
   ---
   
   ## âš ī¸ Medium Issues
   
   ### 1. Integer Overflow in `zstd_compress_operation()`
   
   **Location**: `plugins/compress/compress.cc:514`
   
   **Problem**: Casting `int64_t upstream_length` to `size_t` without bounds checking. If `upstream_length` is negative (malicious input or bug), the cast wraps to a huge positive value.
   
   **Current Code**:
   ```cpp
   ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};
   ```
   
   **Recommended Fix**:
   ```cpp
   if (upstream_length < 0) {
     error("Invalid upstream_length: %" PRId64 " (negative value)", upstream_length);
     return false;
   }
   
   if (upstream_buffer == nullptr && upstream_length > 0) {
     error("upstream_buffer is NULL with non-zero length");
     return false;
   }
   
   ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};
   ```
   
   ### 2. Similar Issue with `downstream_length`
   
   **Location**: `plugins/compress/compress.cc:520`
   
   **Problem**: Same cast issue, though less critical since it comes from ATS API.
   
   **Recommended**: Add validation:
   ```cpp
   if (downstream_length < 0) {
     error("Invalid downstream_length from TSIOBufferBlockWriteStart");
     return false;
   }
   ```
   
   ---
   
   ## â„šī¸ Minor Issues
   
   ### 1. Typo in Comment
   
   **Location**: `plugins/compress/compress.cc:236`
   
   **Issue**: Comment says "brotlidestory" instead of "brotli destroy"
   
   **Fix**: Change to `// Brotli destroy`
   
   ### 2. Inconsistent Error Logging
   
   **Location**: `plugins/compress/compress.cc:395-398`
   
   **Issue**: Mix of `TSError()` and `error()` macro.
   
   **Current**:
   ```cpp
   if (!data->zstrm_zstd.cctx) {
     TSError("Failed to create Zstandard compression context");  // Uses TSError
     return;
   }
   ```
   
   **Elsewhere**:
   ```cpp
   error("Failed to initialize Zstd compression context");  // Uses error()
   ```
   
   **Recommendation**: Use consistent error logging throughout the ZSTD code - prefer `error()` macro for consistency with the rest of the file.
   
   ---
   
   ## ✅ Excellent Practices Observed
   
   1. **Proper error checking**: All ZSTD errors are checked with `ZSTD_isError()` and error names are logged
   2. **Resource cleanup**: `ZSTD_freeCCtx()` is properly called in `data_destroy()`
   3. **Data integrity**: Checksum is enabled with `ZSTD_c_checksumFlag`
   4. **Infinite loop prevention**: Code checks for no progress in compression loop (line 542-544)
   5. **Proper compression modes**: Uses `ZSTD_e_continue` and `ZSTD_e_flush` appropriately
   6. **Backward compatibility**: Gracefully handles missing ZSTD libraries
   
   ---
   
   ## Summary
   
   - **Critical Issues**: 1 (must fix before merge)
   - **Medium Issues**: 2 (should fix for robustness)
   - **Minor Issues**: 2 (nice to fix)
   
   **Overall Assessment**: The ZSTD implementation is well-structured and follows the ZSTD API correctly. The critical initialization issue should be addressed, and the integer overflow checks would significantly improve robustness. Great work on achieving excellent compression ratios and maintaining backward compatibility!
   
   **Testing**: All three test phases passed successfully ✅
   - Test 1 (Master baseline): GZIP and Brotli working
   - Test 2 (PR without ZSTD): No regressions, graceful fallback
   - Test 3 (PR with ZSTD): ZSTD achieves 98.6% compression (4701 → 68 bytes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3503589033

   It can be done in a followup PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2280700576


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -192,12 +192,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!
+
+============== =================================================================
+Algorithm      Description
+============== =================================================================
+gzip           Standard gzip compression (default, widely supported)
+deflate        Deflate compression (RFC 1951)
+br             Brotli compression (modern, efficient)
+zstd           Zstandard compression (fast, high compression ratio)

Review Comment:
   Great summary. Thank you for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2281755612


##########
CMakeLists.txt:
##########
@@ -341,6 +341,11 @@ set(TS_USE_MALLOC_ALLOCATOR ${ENABLE_MALLOC_ALLOCATOR})
 set(TS_USE_ALLOCATOR_METRICS ${ENABLE_ALLOCATOR_METRICS})
 find_package(ZLIB REQUIRED)
 
+find_package(zstd)
+if(zstd_FOUND)
+  set(HAVE_ZSTD_H TRUE)
+endif()

Review Comment:
   thank you for this suggestion, I have now done similar in [`91029c5` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/91029c557572da3b6c45cc6170924609ed3ee439)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2056285205


##########
plugins/compress/compress.cc:
##########
@@ -233,6 +255,9 @@ content_encoding_header(TSMBuffer bufp, TSMLoc hdr_loc, const int compression_ty
   } else if (compression_type & COMPRESSION_TYPE_DEFLATE && (algorithm & ALGORITHM_DEFLATE)) {
     value     = TS_HTTP_VALUE_DEFLATE;
     value_len = TS_HTTP_LEN_DEFLATE;
+  } else if (compression_type & COMPRESSION_TYPE_ZSTD && (algorithm & ALGORITHM_ZSTD)) {
+    value     = "zstd";
+    value_len = strlen("zstd");

Review Comment:
   Note to self: These should be moved into TS_HTTP_VALUE_ZSTD and TS_HTTP_LEN_ZSTD



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-2816727717

   @masaori335 do you know which files I need to modify to install zstd on the osx used in the Jenkins job?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2284390160


##########
src/traffic_layout/info.cc:
##########
@@ -203,6 +212,11 @@ produce_versions(bool json)
 #else
   print_var("brotli", undef, json);
 #endif
+#if HAVE_ZSTD_H
+  print_var("zstd", LBW().print("{}", ZSTD_versionString()).view(), json);
+#else
+  print_var("zstd", undef, json);
+#endif

Review Comment:
   thank you - updated in [`4608bad` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/4608badde2130690bd72e4c08a7b0f5abf4bafe7)



##########
src/traffic_layout/info.cc:
##########
@@ -91,6 +95,11 @@ produce_features(bool json)
 #else
   print_feature("TS_HAS_BROTLI", 0, json);
 #endif
+#if HAVE_ZSTD_H
+  print_feature("TS_HAS_ZSTD", 1, json);
+#else
+  print_feature("TS_HAS_ZSTD", 0, json);
+#endif

Review Comment:
   thank you - updated in [`4608bad` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/4608badde2130690bd72e4c08a7b0f5abf4bafe7)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3498717563

   ## Fedora 43 Build Support
   
   For Fedora 43 (and potentially other RHEL-based systems), the zstd package doesn't ship CMake config files, only pkg-config files. Here's a suggested enhancement to add pkg-config fallback support:
   
   ```diff
   diff --git a/CMakeLists.txt b/CMakeLists.txt
   index 692b11985a..0e1d26e08f 100644
   --- a/CMakeLists.txt
   +++ b/CMakeLists.txt
   @@ -365,7 +365,20 @@ if(zstd_FOUND)
        endif()
      endif()
    else()
   -  set(HAVE_ZSTD_H FALSE)
   +  # Try pkg-config as fallback
   +  find_package(PkgConfig QUIET)
   +  if(PKG_CONFIG_FOUND)
   +    pkg_check_modules(ZSTD QUIET IMPORTED_TARGET libzstd)
   +    if(ZSTD_FOUND)
   +      add_library(zstd::zstd ALIAS PkgConfig::ZSTD)
   +      set(HAVE_ZSTD_H TRUE)
   +      message(STATUS "Found zstd via pkg-config: ${ZSTD_VERSION}")
   +    else()
   +      set(HAVE_ZSTD_H FALSE)
   +    endif()
   +  else()
   +    set(HAVE_ZSTD_H FALSE)
   +  endif()
    endif()
    
    # ncurses is used in traffic_top
   ```
   
   This allows the build to find zstd on distributions that only provide `libzstd.pc` without CMake package config files. Tested successfully on Fedora 43 with libzstd-devel 1.5.7.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bneradt (via GitHub)" <gi...@apache.org>.
bneradt commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2467144290


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   Oh, I see. I thought it was a dev comment about how to write the doc. But it's referring to the list. Yes, please ignore my comment. Thank you @JakeChampion and @maskit . 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "maskit (via GitHub)" <gi...@apache.org>.
maskit commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2319465520


##########
tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml:
##########
@@ -784,11 +812,6 @@ sessions:
         - [ X-Response-Identifier, { value: Br-Accept-Encoding, as: equal } ]
         - [ X-Cache, { value: hit-fresh, as: equal } ]
 
-  # NOTICE: This case should make Accept-Encoding header br, gzip;q=0.8 match
-  #         the alternate of br, gzip Accept-Encoding header.
-  #         However, due to the implementation of calculate_quality_of_match(),
-  #         ATS matches the alternate of gzip Accept-Encoding header.
-  #         The result is DIFFERENT from the description of proxy.config.http.normalize_ae: 3

Review Comment:
   This test code was provided by another contributor, and obviously I missed this notice.
   
   It looks like the contributor found a bug in a place they didn't touch, and kept it as it was. The lines commented out must be the true expected results. Since this PR uncomment those, I suppose the bug was fixed on this PR as well.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3193903233

   @cmcfarlen do you know if there's anything I can do to help get this reviewed? I'm happy to join a mailing list or slack, discord, etc - my work uses ATS quite a bit and I'm eager to ensure we contribute back to the project where we can


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2281743615


##########
include/tscore/ink_config.h.cmake.in:
##########
@@ -187,3 +187,5 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@;
 #cmakedefine YAMLCPP_LIB_VERSION "@YAMLCPP_LIB_VERSION@"
 
 #cmakedefine01 TS_HAS_CRIPTS
+
+#cmakedefine HAVE_ZSTD_H 1

Review Comment:
   The intention is for the feature to be toggled via the compress plugin options, similar to how the compress plugin introduced brotli compression support.
   If someone configures the compress plugin to use zstd and they do not have it available, it will error with:
   >error("supported-algorithms: zstd support not compiled in.");
   
   from https://github.com/apache/trafficserver/pull/12201/files#diff-908cc31df4fc1801284333e04798e2c8a9b1f4e2cb31ca97bb16f24839e0ac28R236-R241



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3071587727

   @bryancall i think this is ready to review now 🙏 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2486649315


##########
plugins/compress/configuration.cc:
##########
@@ -208,7 +218,7 @@ HostConfiguration::add_compression_algorithms(swoc::TextView line)
     } else if (token == "deflate") {
       compression_algorithms_ |= ALGORITHM_DEFLATE;
     } else {
-      error("Unknown compression type. Supported compression-algorithms <br,gzip,deflate>.");
+      error("Unknown compression type. Supported compression-algorithms <zstd,br,gzip,deflate>.");

Review Comment:
   done in [`8e4f9c8` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/8e4f9c844dd4e986f07f16467580266f0bffeaf4)



##########
plugins/compress/compress.cc:
##########
@@ -467,6 +504,144 @@ brotli_transform_one(Data *data, const char *upstream_buffer, int64_t upstream_l
 }
 #endif
 
+#if HAVE_ZSTD_H
+static bool
+zstd_compress_operation(Data *data, const char *upstream_buffer, int64_t upstream_length, ZSTD_EndDirective mode)
+{
+  TSIOBufferBlock downstream_blkp;
+  int64_t         downstream_length;
+
+  ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};
+
+  for (;;) {
+    downstream_blkp         = TSIOBufferStart(data->downstream_buffer);
+    char *downstream_buffer = TSIOBufferBlockWriteStart(downstream_blkp, &downstream_length);
+
+    ZSTD_outBuffer output = {downstream_buffer, static_cast<size_t>(downstream_length), 0};
+
+    size_t result = ZSTD_compressStream2(data->zstrm_zstd.cctx, &output, &input, mode);
+
+    if (ZSTD_isError(result)) {
+      error("Zstd compression failed (%d): %s", mode, ZSTD_getErrorName(result));
+      return false;
+    }
+
+    if (output.pos > 0) {
+      TSIOBufferProduce(data->downstream_buffer, output.pos);
+      data->downstream_length    += output.pos;
+      data->zstrm_zstd.total_out += output.pos;
+    }
+
+    // Check completion conditions based on mode
+    if (mode == ZSTD_e_continue) {
+      // For continue mode, stop when all input is consumed
+      if (input.pos >= input.size) {
+        break;
+      }
+      // If we have output space but no more input was consumed, break to avoid infinite loop
+      if (output.pos == 0 && input.pos < input.size) {
+        error("zstd-transform: no progress made in compression");
+        return false;
+      }
+    } else if (mode == ZSTD_e_flush) {
+      // For flush mode, stop when flush is complete (result == 0)
+      if (result == 0) {
+        break;
+      }
+    }
+  }
+
+  return true;
+}
+
+static void
+zstd_transform_init(Data *data)

Review Comment:
   done in [`e03fa5c` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/e03fa5cf656c1e5230ce0071ed24e34cb92d3d2a)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2486650535


##########
plugins/compress/compress.cc:
##########
@@ -467,6 +504,144 @@ brotli_transform_one(Data *data, const char *upstream_buffer, int64_t upstream_l
 }
 #endif
 
+#if HAVE_ZSTD_H
+static bool
+zstd_compress_operation(Data *data, const char *upstream_buffer, int64_t upstream_length, ZSTD_EndDirective mode)
+{
+  TSIOBufferBlock downstream_blkp;
+  int64_t         downstream_length;
+
+  ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};

Review Comment:
   good catch! fixed in [`b87c864` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/b87c8645cdba70aff9f104c97b38ebaacf924b27)



##########
plugins/compress/compress.cc:
##########
@@ -467,6 +504,144 @@ brotli_transform_one(Data *data, const char *upstream_buffer, int64_t upstream_l
 }
 #endif
 
+#if HAVE_ZSTD_H
+static bool
+zstd_compress_operation(Data *data, const char *upstream_buffer, int64_t upstream_length, ZSTD_EndDirective mode)
+{
+  TSIOBufferBlock downstream_blkp;
+  int64_t         downstream_length;
+
+  ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};
+
+  for (;;) {
+    downstream_blkp         = TSIOBufferStart(data->downstream_buffer);
+    char *downstream_buffer = TSIOBufferBlockWriteStart(downstream_blkp, &downstream_length);
+
+    ZSTD_outBuffer output = {downstream_buffer, static_cast<size_t>(downstream_length), 0};

Review Comment:
   completed in [`1b1ed1c` (#12201)](https://github.com/apache/trafficserver/pull/12201/commits/1b1ed1c54c599818ab38ee6ee043988868ce01e1)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3474668732

   ## Comprehensive Testing Results for ZSTD Compression Support
   
   I've completed comprehensive testing of PR #12201 across three test phases. All tests **PASSED** ✅
   
   ### Test Environment
   - Build: `dev-asan` preset
   - Test file: 4701 bytes HTML
   - Configuration: `proxy.config.http.normalize_ae: 0` (required for Brotli/ZSTD)
   
   ---
   
   ### 📊 Test Results Summary
   
   #### **Test 1: Baseline (Master Branch)**
   Established baseline functionality with existing compression algorithms:
   - ✅ **GZIP**: 4701 → 91 bytes (98.1% reduction)
   - ✅ **Brotli**: 4701 → 53 bytes (98.9% reduction)
   
   #### **Test 2: PR #12201 WITHOUT ZSTD Libraries**
   Verified backward compatibility when ZSTD libraries are not available:
   - ✅ **GZIP**: 4701 → 91 bytes (98.1% reduction)
   - ✅ **Brotli**: 4701 → 53 bytes (98.9% reduction)
   - ✅ **ZSTD**: 4701 bytes (uncompressed - graceful fallback)
   
   **Result**: No regressions. Existing compression algorithms continue to work when ZSTD is unavailable.
   
   #### **Test 3: PR #12201 WITH ZSTD Libraries**
   Verified new ZSTD compression functionality:
   - ✅ **GZIP**: 4701 → 91 bytes (98.1% reduction)
   - ✅ **Brotli**: 4701 → 53 bytes (98.9% reduction)
   - ✅ **ZSTD**: 4701 → 68 bytes (98.6% reduction) ⭐ **NEW!**
   
   **Result**: ZSTD compression works correctly and produces valid Zstandard compressed data.
   
   ---
   
   ### đŸŽ¯ Key Findings
   
   1. **✅ ZSTD compression works correctly** - Produces valid compressed output with excellent compression ratios
   2. **✅ Backward compatibility maintained** - PR works with and without ZSTD libraries installed
   3. **✅ No regressions** - Existing GZIP and Brotli compression continue to work perfectly
   4. **✅ Graceful fallback** - When ZSTD libraries are unavailable, requests are served uncompressed without errors
   
   ### 📈 Compression Algorithm Performance
   
   On the 4701-byte test file:
   1. **Brotli**: 53 bytes (best compression)
   2. **ZSTD**: 68 bytes (+28% vs Brotli)
   3. **GZIP**: 91 bytes (+72% vs Brotli)
   
   ZSTD achieves excellent compression, sitting between Brotli and GZIP in terms of compression ratio.
   
   ---
   
   ### âš™ī¸ Configuration Requirements
   
   **Critical**: For Brotli and ZSTD to function properly, set:
   ```yaml
   records:
     http:
       normalize_ae: 0
   ```
   
   Without this setting, ATS core normalizes the `Accept-Encoding` header before the compress plugin processes it, which interferes with Brotli/ZSTD compression.
   
   ---
   
   ### đŸ—ī¸ Build Notes
   
   The PR's ZSTD detection via `find_package(zstd CONFIG)` requires a CMake config file. On systems where `libzstd-devel` doesn't provide `zstdConfig.cmake`, a simple config file can be created:
   
   ```cmake
   add_library(zstd::zstd UNKNOWN IMPORTED)
   set_target_properties(zstd::zstd PROPERTIES
     IMPORTED_LOCATION "/usr/lib64/libzstd.so"
     INTERFACE_INCLUDE_DIRECTORIES "/usr/include"
   )
   set(zstd_FOUND TRUE)
   ```
   
   ---
   
   ### ✅ Conclusion
   
   PR #12201 successfully implements ZSTD compression support without any regressions. The implementation:
   - Maintains full backward compatibility
   - Handles missing ZSTD libraries gracefully
   - Achieves excellent compression ratios (98.6% reduction)
   - Integrates seamlessly with existing compression infrastructure
   
   All three test phases passed successfully. Great work on this feature! 🎉
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "zwoop (via GitHub)" <gi...@apache.org>.
zwoop commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2511992539


##########
src/iocore/cache/HttpTransactCache.cc:
##########
@@ -1244,7 +1244,11 @@ HttpTransactCache::CalcVariability(const HttpConfigAccessor *http_config_params,
 
         // Disable Vary mismatch checking for Accept-Encoding.  This is only safe to
         // set if you are promising to fix any Accept-Encoding/Content-Encoding mismatches.
-        if (http_config_params->get_ignore_accept_encoding_mismatch() &&
+        // Only suppress variability checks when the operator explicitly set

Review Comment:
   This feels like a bug that's been here for a long time. @cmcfarlen @ezelkow1 I wonder if we should consider this for back ports to 10.x and even 9.2 ? Only this portion though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "zwoop (via GitHub)" <gi...@apache.org>.
zwoop commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2512007497


##########
plugins/compress/compress.cc:
##########
@@ -580,7 +620,14 @@ transformable(TSHttpTxn txnp, bool server, HostConfiguration *host_configuration
         continue;
       }
 
-      if (strncasecmp(value, "br", sizeof("br") - 1) == 0) {
+      info("Accept-Encoding value [%.*s]", len, value);

Review Comment:
   Nit picking (and I know it's the same), but this feels more like a debug() than an info(). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-2820475315

   > We need to ask @ezelkow1 to install the package in the osx env, I guess.
   > 
   > However, this PR has `#if HAVE_ZSTD_F` check, why it's failing?
   
   thank you, _F was a mistake, it was meant to be _H


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2280699760


##########
ci/docker/deb/Dockerfile:
##########
@@ -55,7 +55,8 @@ RUN apt-get update; apt-get -y dist-upgrade; \
     apt-get -y install libssl-dev libexpat1-dev libpcre3-dev libcap-dev \
     libhwloc-dev libunwind8 libunwind-dev zlib1g-dev \
     tcl-dev tcl8.6-dev libjemalloc-dev libluajit-5.1-dev liblzma-dev \
-    libhiredis-dev libbrotli-dev libncurses-dev libgeoip-dev libmagick++-dev; \
+    libhiredis-dev libbrotli-dev libncurses-dev libgeoip-dev libmagick++-dev \

Review Comment:
   What's the rationale for including this optional dependency in the docker images by default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2280700254


##########
cmake/Findzstd.cmake:
##########
@@ -0,0 +1,53 @@
+#######################

Review Comment:
   From a quick look at the zstd GitHub page, it looks like at least the most recent version has a CMake config that I would hope gets included in the packages for each distro. Is it still necessary to write our own find script?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bneradt (via GitHub)" <gi...@apache.org>.
bneradt commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2319277122


##########
tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml:
##########
@@ -784,11 +812,6 @@ sessions:
         - [ X-Response-Identifier, { value: Br-Accept-Encoding, as: equal } ]
         - [ X-Cache, { value: hit-fresh, as: equal } ]
 
-  # NOTICE: This case should make Accept-Encoding header br, gzip;q=0.8 match
-  #         the alternate of br, gzip Accept-Encoding header.
-  #         However, due to the implementation of calculate_quality_of_match(),
-  #         ATS matches the alternate of gzip Accept-Encoding header.
-  #         The result is DIFFERENT from the description of proxy.config.http.normalize_ae: 3

Review Comment:
   Can you help me understand why this comment is being removed? Is the normalize_ae 3 alternate matching behavior changing?
   
   Tagging @maskit / @moonchen since I think they worked with this test in the past.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2283664657


##########
plugins/compress/compress.cc:
##########
@@ -136,8 +132,15 @@ handle_range_request(TSMBuffer req_buf, TSMLoc req_loc, HostConfiguration *hc)
 }
 } // namespace
 
+// Forward declarations for ZSTD compression functions
+#if HAVE_ZSTD_H
+static void zstd_compress_init(Data *data);
+static void zstd_compress_finish(Data *data);
+static void zstd_compress_one(Data *data, const char *upstream_buffer, int64_t upstream_length);

Review Comment:
   I'm curious why the name is `zstd_compress_one` and not `zstd_compress_update`. I worked on the hashing before and this reminds me of OpenSSL's API, except for that naming difference.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3474755275

   I'll make the changes in this pr if that's okay 👍


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2463133342


##########
tests/gold_tests/pluginTest/compress/compress.gold:
##########
@@ -1,22 +1,22 @@
-> GET ``/obj0 HTTP/1.1
-> X-Ats-Compress-Test: 0/gzip, deflate, sdch, br
-> Accept-Encoding: gzip, deflate, sdch, br
+> GET http://ae-0/obj0 HTTP/1.1
+> X-Ats-Compress-Test: 0/gzip, deflate, sdch, br, zstd
+> Accept-Encoding: gzip, deflate, sdch, br, zstd
 < HTTP/1.1 200 OK
 < Content-Type: text/javascript
 < Content-Encoding: br
 < Vary: Accept-Encoding
 < Content-Length: 46
 ===
-> GET ``/obj0 HTTP/1.1
+> GET http://ae-0/obj0 HTTP/1.1
 > X-Ats-Compress-Test: 0/gzip
 > Accept-Encoding: gzip
 < HTTP/1.1 200 OK
 < Content-Type: text/javascript
 < Content-Encoding: gzip
 < Vary: Accept-Encoding
-< Content-Length: 7``
+< Content-Length: 72

Review Comment:
   This change is causing autest to fail for me on my Mac.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2436746206


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   I'll do that in a new pr 👍



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "masaori335 (via GitHub)" <gi...@apache.org>.
masaori335 commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-2817381449

   We need to ask @ezelkow1 to install the package in the osx env, I guess. 
   
   However, this PR has `#if HAVE_ZSTD_F` check, why it's failing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-2835881366

   Hi! This doesn't compile for me.  I added a line to `include/tscore/ink_config.cmake.in` so `HAVE_ZSTD_H` would be truthy for the CPP macros.  This lit up that code, but then there were several compiler errors:
   
   ```
   /Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/compress.cc:203:9: error: no member named 'zstd_ctx' in 'Data'; did you mean 'zstd_cctx'?
     203 |   data->zstd_ctx = nullptr;
         |         ^~~~~~~~
         |         zstd_cctx
   /Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/misc.h:93:14: note: 'zstd_cctx' declared here
      93 |   ZSTD_CCtx *zstd_cctx;
         |              ^
   /Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/compress.cc:206:11: error: no member named 'zstd_ctx' in 'Data'; did you mean 'zstd_cctx'?
     206 |     data->zstd_ctx = ZSTD_createCCtx();
         |           ^~~~~~~~
         |           zstd_cctx
   /Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/misc.h:93:14: note: 'zstd_cctx' declared here
      93 |   ZSTD_CCtx *zstd_cctx;
         |              ^
   /Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/compress.cc:207:16: error: no member named 'zstd_ctx' in 'Data'; did you mean 'zstd_cctx'?
     207 |     if (!data->zstd_ctx) {
         |                ^~~~~~~~
         |                zstd_cctx
   /Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/misc.h:93:14: note: 'zstd_cctx' declared here
      93 |   ZSTD_CCtx *zstd_cctx;
         |              ^
   ```
   
   Please include this patch in your PR to enable the ZSTD code.  Thanks!
   
   ```
   diff --git a/include/tscore/ink_config.h.cmake.in b/include/tscore/ink_config.h.cmake.in
   index 634ed94c3..7da0771fd 100644
   --- a/include/tscore/ink_config.h.cmake.in
   +++ b/include/tscore/ink_config.h.cmake.in
   @@ -184,3 +184,5 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@;
    #cmakedefine YAMLCPP_LIB_VERSION "@YAMLCPP_LIB_VERSION@"
   
    #cmakedefine01 TS_HAS_CRIPTS
   +
   +#cmakedefine HAVE_ZSTD_H 1
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bneradt (via GitHub)" <gi...@apache.org>.
bneradt commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2429660884


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   haha, I actuallly didn't realize that was in there originally. Good point.
   
   Yes, let's remove it if you don't mind. It's not very user-friendly to contain a dev-only comment in the published docs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3498963027

   > ## Fedora 43 Build Support
   > 
   > For Fedora 43 (and potentially other RHEL-based systems), the zstd package doesn't ship CMake config files, only pkg-config files. Here's a suggested enhancement to add pkg-config fallback support:
   > 
   > ```diff
   > diff --git a/CMakeLists.txt b/CMakeLists.txt
   > index 692b11985a..0e1d26e08f 100644
   > --- a/CMakeLists.txt
   > +++ b/CMakeLists.txt
   > @@ -365,7 +365,20 @@ if(zstd_FOUND)
   >      endif()
   >    endif()
   >  else()
   > -  set(HAVE_ZSTD_H FALSE)
   > +  # Try pkg-config as fallback
   > +  find_package(PkgConfig QUIET)
   > +  if(PKG_CONFIG_FOUND)
   > +    pkg_check_modules(ZSTD QUIET IMPORTED_TARGET libzstd)
   > +    if(ZSTD_FOUND)
   > +      add_library(zstd::zstd ALIAS PkgConfig::ZSTD)
   > +      set(HAVE_ZSTD_H TRUE)
   > +      message(STATUS "Found zstd via pkg-config: ${ZSTD_VERSION}")
   > +    else()
   > +      set(HAVE_ZSTD_H FALSE)
   > +    endif()
   > +  else()
   > +    set(HAVE_ZSTD_H FALSE)
   > +  endif()
   >  endif()
   >  
   >  # ncurses is used in traffic_top
   > ```
   > 
   > This allows the build to find zstd on distributions that only provide `libzstd.pc` without CMake package config files. Tested successfully on Fedora 43 with libzstd-devel 1.5.7.
   
   @bryancall could this be done in a follow up or would we want this as part of the current pull request?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2284387289


##########
plugins/compress/compress.cc:
##########
@@ -136,8 +132,15 @@ handle_range_request(TSMBuffer req_buf, TSMLoc req_loc, HostConfiguration *hc)
 }
 } // namespace
 
+// Forward declarations for ZSTD compression functions
+#if HAVE_ZSTD_H
+static void zstd_compress_init(Data *data);
+static void zstd_compress_finish(Data *data);
+static void zstd_compress_one(Data *data, const char *upstream_buffer, int64_t upstream_length);

Review Comment:
   It looks to be a convention used in this plugin (which I seemed to have half followed...)
   These exist:
   - gzip_transform_one
   - gzip_transform_finish
   - brotli_transform_one
   - brotli_transform_finish
   - brotli_compress_operation
   
   Perhaps I should rename the zstd function to
   - zstd_transform_init
   - zstd_transform_finish
   - zstd_transform_one
   
   what do you think?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2280699295


##########
CMakeLists.txt:
##########
@@ -341,6 +341,11 @@ set(TS_USE_MALLOC_ALLOCATOR ${ENABLE_MALLOC_ALLOCATOR})
 set(TS_USE_ALLOCATOR_METRICS ${ENABLE_ALLOCATOR_METRICS})
 find_package(ZLIB REQUIRED)
 
+find_package(zstd)
+if(zstd_FOUND)
+  set(HAVE_ZSTD_H TRUE)
+endif()

Review Comment:
   This is probably fine, but it does leave `HAVE_ZSTD_H` unset if the condition is false. What do you think about the following?
   ```suggestion
   set(HAVE_ZSTD_H ${zstd_FOUND})
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2482539158


##########
plugins/compress/configuration.cc:
##########
@@ -208,7 +218,7 @@ HostConfiguration::add_compression_algorithms(swoc::TextView line)
     } else if (token == "deflate") {
       compression_algorithms_ |= ALGORITHM_DEFLATE;
     } else {
-      error("Unknown compression type. Supported compression-algorithms <br,gzip,deflate>.");
+      error("Unknown compression type. Supported compression-algorithms <zstd,br,gzip,deflate>.");

Review Comment:
   Looks like this error should be conditional based on how the plugin was compiled



##########
plugins/compress/compress.cc:
##########
@@ -467,6 +504,144 @@ brotli_transform_one(Data *data, const char *upstream_buffer, int64_t upstream_l
 }
 #endif
 
+#if HAVE_ZSTD_H
+static bool
+zstd_compress_operation(Data *data, const char *upstream_buffer, int64_t upstream_length, ZSTD_EndDirective mode)
+{
+  TSIOBufferBlock downstream_blkp;
+  int64_t         downstream_length;
+
+  ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};
+
+  for (;;) {
+    downstream_blkp         = TSIOBufferStart(data->downstream_buffer);
+    char *downstream_buffer = TSIOBufferBlockWriteStart(downstream_blkp, &downstream_length);
+
+    ZSTD_outBuffer output = {downstream_buffer, static_cast<size_t>(downstream_length), 0};
+
+    size_t result = ZSTD_compressStream2(data->zstrm_zstd.cctx, &output, &input, mode);
+
+    if (ZSTD_isError(result)) {
+      error("Zstd compression failed (%d): %s", mode, ZSTD_getErrorName(result));
+      return false;
+    }
+
+    if (output.pos > 0) {
+      TSIOBufferProduce(data->downstream_buffer, output.pos);
+      data->downstream_length    += output.pos;
+      data->zstrm_zstd.total_out += output.pos;
+    }
+
+    // Check completion conditions based on mode
+    if (mode == ZSTD_e_continue) {
+      // For continue mode, stop when all input is consumed
+      if (input.pos >= input.size) {
+        break;
+      }
+      // If we have output space but no more input was consumed, break to avoid infinite loop
+      if (output.pos == 0 && input.pos < input.size) {
+        error("zstd-transform: no progress made in compression");
+        return false;
+      }
+    } else if (mode == ZSTD_e_flush) {
+      // For flush mode, stop when flush is complete (result == 0)
+      if (result == 0) {
+        break;
+      }
+    }
+  }
+
+  return true;
+}
+
+static void
+zstd_transform_init(Data *data)

Review Comment:
   How do we know if the context is partially initialized?  Seems like we should return a bool and check the return value.



##########
contrib/docker/ubuntu/noble/Dockerfile:
##########
@@ -48,6 +48,9 @@ RUN apt update \
     libpcre3-dev \
     hwloc \
     libbrotli-dev \
+    libzstd-dev \
+    luajit \
+    luajit \

Review Comment:
   Should only be listed 1 time (luajit)



##########
plugins/compress/compress.cc:
##########
@@ -467,6 +504,144 @@ brotli_transform_one(Data *data, const char *upstream_buffer, int64_t upstream_l
 }
 #endif
 
+#if HAVE_ZSTD_H
+static bool
+zstd_compress_operation(Data *data, const char *upstream_buffer, int64_t upstream_length, ZSTD_EndDirective mode)
+{
+  TSIOBufferBlock downstream_blkp;
+  int64_t         downstream_length;
+
+  ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};

Review Comment:
   No check to see if upstream_length is positive, can cause int overflow issue.



##########
plugins/compress/compress.cc:
##########
@@ -467,6 +504,144 @@ brotli_transform_one(Data *data, const char *upstream_buffer, int64_t upstream_l
 }
 #endif
 
+#if HAVE_ZSTD_H
+static bool
+zstd_compress_operation(Data *data, const char *upstream_buffer, int64_t upstream_length, ZSTD_EndDirective mode)
+{
+  TSIOBufferBlock downstream_blkp;
+  int64_t         downstream_length;
+
+  ZSTD_inBuffer input = {upstream_buffer, static_cast<size_t>(upstream_length), 0};
+
+  for (;;) {
+    downstream_blkp         = TSIOBufferStart(data->downstream_buffer);
+    char *downstream_buffer = TSIOBufferBlockWriteStart(downstream_blkp, &downstream_length);
+
+    ZSTD_outBuffer output = {downstream_buffer, static_cast<size_t>(downstream_length), 0};

Review Comment:
   downstream_length can be an int overflow here too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2281723070


##########
ci/docker/deb/Dockerfile:
##########
@@ -55,7 +55,8 @@ RUN apt-get update; apt-get -y dist-upgrade; \
     apt-get -y install libssl-dev libexpat1-dev libpcre3-dev libcap-dev \
     libhwloc-dev libunwind8 libunwind-dev zlib1g-dev \
     tcl-dev tcl8.6-dev libjemalloc-dev libluajit-5.1-dev liblzma-dev \
-    libhiredis-dev libbrotli-dev libncurses-dev libgeoip-dev libmagick++-dev; \
+    libhiredis-dev libbrotli-dev libncurses-dev libgeoip-dev libmagick++-dev \

Review Comment:
   The intention was to make it simpler to try out the new feature by ensuring the images contained the dependency already



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2281877062


##########
include/tscore/ink_config.h.cmake.in:
##########
@@ -187,3 +187,5 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@;
 #cmakedefine YAMLCPP_LIB_VERSION "@YAMLCPP_LIB_VERSION@"
 
 #cmakedefine01 TS_HAS_CRIPTS
+
+#cmakedefine HAVE_ZSTD_H 1

Review Comment:
   I see, Brotli and LZMA use this naming scheme as well, so you are right.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall merged PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2284925830


##########
plugins/compress/compress.cc:
##########
@@ -136,8 +132,15 @@ handle_range_request(TSMBuffer req_buf, TSMLoc req_loc, HostConfiguration *hc)
 }
 } // namespace
 
+// Forward declarations for ZSTD compression functions
+#if HAVE_ZSTD_H
+static void zstd_compress_init(Data *data);
+static void zstd_compress_finish(Data *data);
+static void zstd_compress_one(Data *data, const char *upstream_buffer, int64_t upstream_length);

Review Comment:
   Following the local convention is good. I think going with the transform language is a great idea for consistency with the rest of the plugin.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2280701768


##########
include/tscore/ink_config.h.cmake.in:
##########
@@ -187,3 +187,5 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@;
 #cmakedefine YAMLCPP_LIB_VERSION "@YAMLCPP_LIB_VERSION@"
 
 #cmakedefine01 TS_HAS_CRIPTS
+
+#cmakedefine HAVE_ZSTD_H 1

Review Comment:
   I think this should be `TS_USE_ZSTD` for consistency with the other optional features and be grouped with them in this file. @cmcfarlen the other optional feature defines use `#cmakedefine01`, should this one as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2320123239


##########
tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml:
##########
@@ -784,11 +812,6 @@ sessions:
         - [ X-Response-Identifier, { value: Br-Accept-Encoding, as: equal } ]
         - [ X-Cache, { value: hit-fresh, as: equal } ]
 
-  # NOTICE: This case should make Accept-Encoding header br, gzip;q=0.8 match
-  #         the alternate of br, gzip Accept-Encoding header.
-  #         However, due to the implementation of calculate_quality_of_match(),
-  #         ATS matches the alternate of gzip Accept-Encoding header.
-  #         The result is DIFFERENT from the description of proxy.config.http.normalize_ae: 3

Review Comment:
   that is correct - i have fixed this bug whilst working on adding zstd to the normalise accept-encoding implementation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "maskit (via GitHub)" <gi...@apache.org>.
maskit commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2430001225


##########
doc/admin-guide/plugins/compress.en.rst:
##########
@@ -202,12 +202,59 @@ supported-algorithms
 
 Provides the compression algorithms that are supported, a comma separate list
 of values. This will allow |TS| to selectively support ``gzip``, ``deflate``,
-and brotli (``br``) compression. The default is ``gzip``. Multiple algorithms can
-be selected using ',' delimiter, for instance, ``supported-algorithms
-deflate,gzip,br``. Note that this list must **not** contain any white-spaces!
+brotli (``br``), and zstd (``zstd``) compression. The default is ``gzip``.
+Multiple algorithms can be selected using ',' delimiter, for instance,
+``supported-algorithms deflate,gzip,br,zstd``. Note that this list must **not**
+contain any white-spaces!

Review Comment:
   Isn't it for plugin users?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2064092924


##########
plugins/compress/compress.cc:
##########
@@ -191,6 +198,16 @@ data_alloc(int compression_type, int compression_algorithms)
     data->bstrm.avail_out = 0;
     data->bstrm.total_out = 0;
   }
+#endif
+#if HAVE_ZSTD_H
+  data->zstd_ctx = nullptr;

Review Comment:
   this should be `data->zstd_cctx` here (as well as other places)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3458849632

   building without zstd enabled does compile, but the compress autest doesn't run:
   ```
   % ./autest.sh -f compress
   Running Test compress: Skipped
   Warning: Skipping test compress because:
    ATS feature not enabled: TS_HAS_ZSTD
   
   Generating Report: --------------
   1 Tests were skipped:
   --------------------------------------------------------------------------------
    Test "compress" Skipped
       File: compress.test.py
       Directory: /Users/cmcfarlen/projects/oss/trafficserver/tests/gold_tests/pluginTest/compress
     Reason: ATS feature not enabled: TS_HAS_ZSTD
   
   Total of 1 test
     Unknown: 0
     Exception: 0
     Failed: 0
     Warning: 0
     Skipped: 1
     Passed: 0
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3506267037

   Apologies for the final commit here which voided the reviews, I saw a code refactor I previously did and realised doesn't need to happen at all so I reverted it, I'll not touch the pr again now 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "JakeChampion (via GitHub)" <gi...@apache.org>.
JakeChampion commented on code in PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#discussion_r2512520332


##########
src/iocore/cache/HttpTransactCache.cc:
##########
@@ -1244,7 +1244,11 @@ HttpTransactCache::CalcVariability(const HttpConfigAccessor *http_config_params,
 
         // Disable Vary mismatch checking for Accept-Encoding.  This is only safe to
         // set if you are promising to fix any Accept-Encoding/Content-Encoding mismatches.
-        if (http_config_params->get_ignore_accept_encoding_mismatch() &&
+        // Only suppress variability checks when the operator explicitly set

Review Comment:
   I have a pr open for this specific change which we can backport 
   https://github.com/apache/trafficserver/pull/12618



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3543507416

   @JakeChampion looks like there is now a conflict and should look at the comment from @zwoop above about info vs debug


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Zstandard compression support and update tests [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #12201:
URL: https://github.com/apache/trafficserver/pull/12201#issuecomment-3607838775

   I changed this feature to only go into ATS 10.2.0.  We are going to branch in January and release in Q1.  We have limited time to test this feature and doing it in one version would be significantly easier for us.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org