[GitHub] [beam] grufino commented on a diff in pull request #22419: Add zstd compression/decompression support

[GitBox <gi...@apache.org>]

grufino commented on code in PR #22419:
URL: https://github.com/apache/beam/pull/22419#discussion_r930043048


##########
sdks/python/apache_beam/io/filesystem.py:
##########
@@ -166,6 +177,9 @@ def _initialize_decompressor(self):
       self._decompressor = bz2.BZ2Decompressor()
     elif self._compression_type == CompressionTypes.DEFLATE:
       self._decompressor = zlib.decompressobj()
+    elif self._compression_type == CompressionTypes.ZSTD:
+      self._decompressor = zstandard.ZstdDecompressor(
+          max_window_size=2147483648).decompressobj()

Review Comment:
   I know this seems arbitrary (and probably is), but without this param the decompressor fails with error: 
   ```
   zstd.ZstdError: zstd decompress error: Frame requires too much memory for decoding
   ```
   I thought about using `DEFAULT_READ_BUFFER_SIZE` but it seems too low for this param. I can either create a constant or a new parameter if necessary, let me know.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org