Re: [PR] [WIP] [Feature] Enable the capability to specify zstd and lz4 segment compression via config [pinot]

["deemoliu (via GitHub)" <gi...@apache.org>]

deemoliu commented on code in PR #14008:
URL: https://github.com/apache/pinot/pull/14008#discussion_r1803900524


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/TarCompressionUtils.java:
##########
@@ -69,6 +69,12 @@ public class TarCompressionUtils {
   private TarCompressionUtils() {
   }
 
+  /**
+   * This generic compressed tar file extension does not bind to a particular compressor. Decompression determines the
+   * appropriate compressor at run-time based on the file's magic number irrespective of the file extension.
+   * Compression uses the default compressor automatically if this generic extension is used.
+   */
+  public static final String TAR_COMPRESSED_FILE_EXTENSION = ".tar.compressed";

Review Comment:
   mostly good. we are able to compress and decompress different segment format.
   
   Curious, if we exposed and updated the value of _defaultCompressorName  (line 86), how can we make sure the `.tar.compressed` files can still be decompress by updated compressor?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org