You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "iemejia (via GitHub)" <gi...@apache.org> on 2026/07/11 13:03:54 UTC
[PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
iemejia opened a new pull request, #3850:
URL: https://github.com/apache/avro/pull/3850
## What is the purpose of the change
The deflate and bzip2 codecs are inflated incrementally and rejected once the output would exceed the limit; snappy rejects an over-large declared length up front; zstandard caps its streaming loop. Exceeding the limit raises `avro.errors.AvroDecompressionSizeException`.
When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size (a decompression bomb). This enforces a configurable maximum decompressed size while reading each block, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the `AVRO_MAX_DECOMPRESS_LENGTH` environment variable.
This is part of the umbrella issue AVRO-4283.
## Verifying this change
This change added tests and can be verified as follows:
- Added `avro/test/test_codecs.py` (`TestDecompressionSizeLimit`) covering deflate and bzip2 over-limit rejection and within-limit decoding (snappy/zstandard covered when those optional deps are installed).
- Run: `cd lang/py && python3 -m unittest avro.test.test_codecs`
## Documentation
- Does this pull request introduce a new feature? (no — hardening/robustness)
- If yes, how is the feature documented? (not applicable; the new `AVRO_MAX_DECOMPRESS_LENGTH` environment variable is documented in code comments)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567571742
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,22 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ if not decompressor.eof:
+ # The end-of-stream marker was not reached: the block is truncated or
+ # corrupt. zlib.decompress() used to raise for this; preserve that.
+ raise avro.errors.InvalidAvroBinaryEncoding("Truncated or corrupt deflate block")
Review Comment:
Good catch — decompress(data, limit+1) leaves unconsumed input, and the subsequent flush() was unbounded. Reworked to drain unconsumed_tail in a loop with a per-call max_length and to pass a bounded length to flush(), so total output can't exceed limit+1 before being rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567341958
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,18 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
Good catch. Added a `decompressor.eof` check after flush() — a truncated/incomplete deflate block now raises `InvalidAvroBinaryEncoding`, restoring the reject-truncated behavior that `zlib.decompress` had.
##########
lang/py/avro/codecs.py:
##########
@@ -158,7 +242,15 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
# Compressed data includes a 4-byte CRC32 checksum
length = readers_decoder.read_long()
data = readers_decoder.read(length - 4)
+ limit = _max_decompress_length()
Review Comment:
Fixed — `SnappyCodec.decompress` now validates `length >= 4` and raises a codec-specific `InvalidAvroBinaryEncoding` before reading `length - 4` bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567557088
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,22 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ if not decompressor.eof:
+ # The end-of-stream marker was not reached: the block is truncated or
+ # corrupt. zlib.decompress() used to raise for this; preserve that.
+ raise avro.errors.InvalidAvroBinaryEncoding("Truncated or corrupt deflate block")
Review Comment:
`DeflateCodec.decompress()` caps `decompress()` output with `max_length`, but `decompressor.flush()` is currently unbounded. If the compressed input expands beyond the limit, the excess output can be materialized during `flush()` (allocating far past the configured cap) before the size check runs, which defeats the decompression-bomb protection goal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3564616023
##########
lang/py/avro/codecs.py:
##########
@@ -139,7 +200,11 @@ def compress(data: bytes) -> Tuple[bytes, int]:
def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
length = readers_decoder.read_long()
data = readers_decoder.read(length)
- uncompressed = bz2.decompress(data)
+ limit = _max_decompress_length()
+ decompressor = bz2.BZ2Decompressor()
+ uncompressed = decompressor.decompress(data, limit + 1)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
return avro.io.BinaryDecoder(io.BytesIO(uncompressed))
Review Comment:
BZip2Codec.decompress now uses bz2.BZ2Decompressor().decompress() once and returns without verifying that the compressed stream reached EOF (or handling concatenated bzip2 streams). This can silently accept truncated/corrupt blocks (and can truncate output if buffered data remains), whereas bz2.decompress() would error on invalid/truncated input and will process concatenated streams. Consider looping until EOF, draining buffered output with empty input, and processing any unused_data as additional streams; raise if input ends before EOF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on PR #3850:
URL: https://github.com/apache/avro/pull/3850#issuecomment-5213777038
Cherry-picked to [branch-1.12|https://github.com/apache/avro/commit/478ed0b433d9bd1db3a2781f8b6b351b9b39d735].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567333194
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,18 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
DeflateCodec now uses zlib.decompressobj with a size cap, but it never verifies that the end-of-stream marker was reached. For truncated/incomplete deflate input, `decompressobj.flush()` can return partial output without raising, which would silently accept corrupt blocks. After flushing, check `decompressor.eof` and raise InvalidAvroBinaryEncoding when false to preserve the previous behavior of rejecting truncated data.
##########
lang/py/avro/codecs.py:
##########
@@ -158,7 +242,15 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
# Compressed data includes a 4-byte CRC32 checksum
length = readers_decoder.read_long()
data = readers_decoder.read(length - 4)
+ limit = _max_decompress_length()
Review Comment:
SnappyCodec.decompress reads `length - 4` bytes for the compressed payload. If the declared block length is < 4 (or negative), this currently falls through to BinaryDecoder.read's generic "expected positive integer" error, which is less actionable and doesn’t clearly identify the codec/block format issue. Validate `length >= 4` and raise InvalidAvroBinaryEncoding with a codec-specific message before attempting the read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3568148828
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +180,31 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Decompress in bounded steps: request at most (limit + 1 - produced)
+ # bytes each call so the accumulated output can never exceed the limit by
+ # more than one byte before being rejected. decompress()'s max_length
+ # leaves unconsumed input in unconsumed_tail, and flush() would otherwise
+ # emit the remainder unbounded, so drain the tail in a loop and bound the
+ # final flush too.
+ uncompressed = bytearray()
+ pending = data
+ while True:
+ want = limit + 1 - len(uncompressed)
+ uncompressed += decompressor.decompress(pending, want)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ pending = decompressor.unconsumed_tail
+ if not pending:
+ break
+ uncompressed += decompressor.flush(limit + 1 - len(uncompressed))
Review Comment:
Fixed — added _decompress_read_ceiling(limit) which returns limit + 1 normally but sys.maxsize when limit == sys.maxsize, so the deflate max_length never overflows Py_ssize_t.
##########
lang/py/avro/codecs.py:
##########
@@ -139,7 +220,30 @@ def compress(data: bytes) -> Tuple[bytes, int]:
def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
length = readers_decoder.read_long()
data = readers_decoder.read(length)
- uncompressed = bz2.decompress(data)
+ limit = _max_decompress_length()
+ uncompressed = bytearray()
+ decompressor = bz2.BZ2Decompressor()
+ input_data = data
+ while True:
+ # Request enough to detect exceeding the limit without allocating
+ # the full (potentially huge) output.
+ want = limit + 1 - len(uncompressed)
+ if want <= 0:
+ want = 1
+ uncompressed += decompressor.decompress(input_data, want)
Review Comment:
Fixed — the bz2 path uses the same _decompress_read_ceiling() so `want` can't become sys.maxsize + 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567421263
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,22 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ if not decompressor.eof:
+ # The end-of-stream marker was not reached: the block is truncated or
+ # corrupt. zlib.decompress() used to raise for this; preserve that.
+ raise avro.errors.InvalidAvroBinaryEncoding("Truncated or corrupt deflate block")
Review Comment:
Added test_deflate_truncated_block_rejected — it truncates a valid deflate block and asserts DeflateCodec.decompress() raises InvalidAvroBinaryEncoding (mirrors the existing bzip2 truncation test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3568128140
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +180,31 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Decompress in bounded steps: request at most (limit + 1 - produced)
+ # bytes each call so the accumulated output can never exceed the limit by
+ # more than one byte before being rejected. decompress()'s max_length
+ # leaves unconsumed input in unconsumed_tail, and flush() would otherwise
+ # emit the remainder unbounded, so drain the tail in a loop and bound the
+ # final flush too.
+ uncompressed = bytearray()
+ pending = data
+ while True:
+ want = limit + 1 - len(uncompressed)
+ uncompressed += decompressor.decompress(pending, want)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ pending = decompressor.unconsumed_tail
+ if not pending:
+ break
+ uncompressed += decompressor.flush(limit + 1 - len(uncompressed))
Review Comment:
When AVRO_MAX_DECOMPRESS_LENGTH is set to a very large value, `_max_decompress_length()` clamps it to `sys.maxsize`, but the deflate path still computes `limit + 1` for the `max_length` argument. If `limit == sys.maxsize`, `limit + 1` overflows `Py_ssize_t` and `zlib` can raise `OverflowError`, defeating the intent of the clamping. Compute a capped `cap` (no `+ 1` when `limit == sys.maxsize`) and ensure `want` stays >= 1 for progress.
##########
lang/py/avro/codecs.py:
##########
@@ -139,7 +220,30 @@ def compress(data: bytes) -> Tuple[bytes, int]:
def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
length = readers_decoder.read_long()
data = readers_decoder.read(length)
- uncompressed = bz2.decompress(data)
+ limit = _max_decompress_length()
+ uncompressed = bytearray()
+ decompressor = bz2.BZ2Decompressor()
+ input_data = data
+ while True:
+ # Request enough to detect exceeding the limit without allocating
+ # the full (potentially huge) output.
+ want = limit + 1 - len(uncompressed)
+ if want <= 0:
+ want = 1
+ uncompressed += decompressor.decompress(input_data, want)
Review Comment:
Same `limit + 1` overflow risk as the deflate path: if the env override is clamped to `sys.maxsize`, `want = limit + 1 - len(uncompressed)` can become `sys.maxsize + 1` and `bz2.BZ2Decompressor.decompress(..., max_length=want)` may raise `OverflowError`. Use a `cap` that avoids `+ 1` at `sys.maxsize`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3564493636
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,16 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output.
+ uncompressed = decompressor.decompress(data, limit + 1)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
Fixed in 638aac57cb — DeflateCodec.decompress now accumulates into a bytearray, so flush() output is appended in place instead of creating an extra full-size copy of the decompressed data.
##########
lang/py/avro/codecs.py:
##########
@@ -184,6 +256,8 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
if not chunk:
break
uncompressed.extend(chunk)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
Fixed in 638aac57cb — the zstandard path now checks `len(uncompressed) + len(chunk) > limit` before extending, so the buffer never grows past the limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567381160
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,22 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ if not decompressor.eof:
+ # The end-of-stream marker was not reached: the block is truncated or
+ # corrupt. zlib.decompress() used to raise for this; preserve that.
+ raise avro.errors.InvalidAvroBinaryEncoding("Truncated or corrupt deflate block")
Review Comment:
The new deflate path now explicitly raises InvalidAvroBinaryEncoding when the end-of-stream marker isn't reached (`not decompressor.eof`). This is an important corruption-rejection behavior, but it's not covered by a unit test (only over-limit/within-limit are tested). Please add a test that builds a valid deflate block, truncates the compressed payload, and asserts `DeflateCodec.decompress()` raises `avro.errors.InvalidAvroBinaryEncoding`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3568059464
##########
lang/py/avro/codecs.py:
##########
@@ -41,6 +42,55 @@
#
STRUCT_CRC32 = struct.Struct(">I") # big-endian unsigned int
+# Name of the environment variable used to override the default maximum size of
+# a single decompressed data-file block.
+MAX_DECOMPRESS_LENGTH_ENV = "AVRO_MAX_DECOMPRESS_LENGTH"
+
+# Default upper bound, in bytes, on the size a single data-file block may
+# decompress to. A block with a very high compression ratio (or a malformed
+# block) can otherwise expand to far more memory than its compressed size.
+# Reading a block that would decompress beyond this limit raises an
+# :class:`avro.errors.AvroDecompressionSizeException`. This mirrors the Java
+# SDK's ``org.apache.avro.limits.decompress.maxLength`` limit (AVRO-4247). The
+# default may be overridden with the ``AVRO_MAX_DECOMPRESS_LENGTH`` environment
+# variable.
+DEFAULT_MAX_DECOMPRESS_LENGTH = 200 * 1024 * 1024 # 200 MiB
+
+
+def _max_decompress_length() -> int:
+ """Return the maximum decompressed block size, honoring the environment override."""
+ value = os.environ.get(MAX_DECOMPRESS_LENGTH_ENV)
+ if value is None:
+ return DEFAULT_MAX_DECOMPRESS_LENGTH
+ try:
+ parsed = int(value)
+ except ValueError:
+ return DEFAULT_MAX_DECOMPRESS_LENGTH
+ return parsed if parsed > 0 else DEFAULT_MAX_DECOMPRESS_LENGTH
Review Comment:
Fixed — _max_decompress_length() now returns `min(parsed, sys.maxsize)` so an oversized override stays a valid Py_ssize_t and won't raise OverflowError inside zlib/bz2 decompress().
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,22 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output. Accumulate into
+ # a bytearray so the flush() output is appended in place rather than
+ # creating an extra full-size copy of the already-decompressed data.
+ uncompressed = bytearray(decompressor.decompress(data, limit + 1))
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ if not decompressor.eof:
+ # The end-of-stream marker was not reached: the block is truncated or
+ # corrupt. zlib.decompress() used to raise for this; preserve that.
+ raise avro.errors.InvalidAvroBinaryEncoding("Truncated or corrupt deflate block")
Review Comment:
Already bounded — the final flush is `decompressor.flush(limit + 1 - len(uncompressed))` (codecs.py:194) followed by the size check, so flush() can't materialize past the cap. The eof check runs after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba merged PR #3850:
URL: https://github.com/apache/avro/pull/3850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3564410918
##########
lang/py/avro/codecs.py:
##########
@@ -123,7 +173,16 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
data = readers_decoder.read_bytes()
# -15 is the log of the window size; negative indicates
# "raw" (no zlib headers) decompression. See zlib.h.
- uncompressed = zlib.decompress(data, -15)
+ limit = _max_decompress_length()
+ decompressor = zlib.decompressobj(-15)
+ # Request at most limit + 1 bytes so that an over-large block is detected
+ # without allocating the whole (potentially huge) output.
+ uncompressed = decompressor.decompress(data, limit + 1)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
+ uncompressed += decompressor.flush()
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
`uncompressed += decompressor.flush()` concatenates immutable `bytes`, creating an extra full-size copy of the already-decompressed data. For large blocks (up to the 200 MiB default limit) this adds significant peak memory and CPU overhead; using a `bytearray` avoids the copy while preserving the size checks.
##########
lang/py/avro/codecs.py:
##########
@@ -184,6 +256,8 @@ def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
if not chunk:
break
uncompressed.extend(chunk)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
Review Comment:
The limit check happens after `uncompressed.extend(chunk)`, so the bytearray can grow past `limit` (by up to one chunk) before raising. Checking `len(uncompressed) + len(chunk)` first avoids allocating beyond the configured cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3564663018
##########
lang/py/avro/codecs.py:
##########
@@ -139,7 +200,11 @@ def compress(data: bytes) -> Tuple[bytes, int]:
def decompress(readers_decoder: avro.io.BinaryDecoder) -> avro.io.BinaryDecoder:
length = readers_decoder.read_long()
data = readers_decoder.read(length)
- uncompressed = bz2.decompress(data)
+ limit = _max_decompress_length()
+ decompressor = bz2.BZ2Decompressor()
+ uncompressed = decompressor.decompress(data, limit + 1)
+ if len(uncompressed) > limit:
+ _raise_decompression_too_large(limit)
return avro.io.BinaryDecoder(io.BytesIO(uncompressed))
Review Comment:
Good catch. Reworked `BZip2Codec.decompress` to verify the decompressor reached end-of-stream and to drain any trailing data, looping over concatenated bzip2 streams so a multi-stream block is fully decoded. Added a test for a truncated block. Pushed in 9f3c49d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4290: [python] Enforce a maximum decompressed block size [avro]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3850:
URL: https://github.com/apache/avro/pull/3850#discussion_r3567608042
##########
lang/py/avro/codecs.py:
##########
@@ -41,6 +42,55 @@
#
STRUCT_CRC32 = struct.Struct(">I") # big-endian unsigned int
+# Name of the environment variable used to override the default maximum size of
+# a single decompressed data-file block.
+MAX_DECOMPRESS_LENGTH_ENV = "AVRO_MAX_DECOMPRESS_LENGTH"
+
+# Default upper bound, in bytes, on the size a single data-file block may
+# decompress to. A block with a very high compression ratio (or a malformed
+# block) can otherwise expand to far more memory than its compressed size.
+# Reading a block that would decompress beyond this limit raises an
+# :class:`avro.errors.AvroDecompressionSizeException`. This mirrors the Java
+# SDK's ``org.apache.avro.limits.decompress.maxLength`` limit (AVRO-4247). The
+# default may be overridden with the ``AVRO_MAX_DECOMPRESS_LENGTH`` environment
+# variable.
+DEFAULT_MAX_DECOMPRESS_LENGTH = 200 * 1024 * 1024 # 200 MiB
+
+
+def _max_decompress_length() -> int:
+ """Return the maximum decompressed block size, honoring the environment override."""
+ value = os.environ.get(MAX_DECOMPRESS_LENGTH_ENV)
+ if value is None:
+ return DEFAULT_MAX_DECOMPRESS_LENGTH
+ try:
+ parsed = int(value)
+ except ValueError:
+ return DEFAULT_MAX_DECOMPRESS_LENGTH
+ return parsed if parsed > 0 else DEFAULT_MAX_DECOMPRESS_LENGTH
Review Comment:
_max_decompress_length() returns the raw int parsed from AVRO_MAX_DECOMPRESS_LENGTH without clamping. If a user sets this to an extremely large value, downstream calls like zlib.decompress(..., max_length=want) / bz2.BZ2Decompressor.decompress(..., max_length=want) can raise OverflowError when converting to a Py_ssize_t, producing a confusing exception instead of cleanly honoring the override (or falling back to the default). Consider clamping to sys.maxsize (and treating non-positive values as default) to keep behavior predictable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org