You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@thrift.apache.org by "Jens-G (via GitHub)" <gi...@apache.org> on 2026/05/27 23:42:20 UTC
[PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Jens-G opened a new pull request, #3547:
URL: https://github.com/apache/thrift/pull/3547
Client: rb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "kpumuk (via GitHub)" <gi...@apache.org>.
kpumuk commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314530019
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against @max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
Review Comment:
Give me some time to benchmark/memory profile this. I am curious if using a string as a result would be more efficient in terms of memory/performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "Jens-G (via GitHub)" <gi...@apache.org>.
Jens-G commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314559488
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against @max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
+ offset = 0
+ begin
+ while offset < compressed.bytesize
+ out = inflater.inflate(compressed.byteslice(offset, ZLIB_INFLATE_CHUNK_SIZE))
Review Comment:
good suggestion, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "kpumuk (via GitHub)" <gi...@apache.org>.
kpumuk commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314637108
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against @max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
Review Comment:
Nope, definitely segfaults and so unsafe. Current implementation is good to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "kpumuk (via GitHub)" <gi...@apache.org>.
kpumuk commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314601407
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against @max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
Review Comment:
For the 16mb field read 1000 times, the results are:
| Variant | Wall | CPU | Peak RSS |
|---|---:|---:|---:|
| pre-commit unbounded `Zlib::Inflate.inflate` | 81.9s | 83.3s | 153 MB |
| array + `join` | 103.5s | 104.0s | 192 MB |
| append with preallocated capacity | 95.7s | 96.2s | 171 MB |
| **append, no preallocation, final patch** | 103.9s | 104.1s | 131 MB |
| single inflate then size check | 80.0s | 81.3s | 169 MB |
I am currently testing shared chunk buffer (caused segfault in my Ruby, trying to figure out if I did it, or I should report it to Ruby) :-)
So far current option (empty string buffer) seems to be the best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "Jens-G (via GitHub)" <gi...@apache.org>.
Jens-G merged PR #3547:
URL: https://github.com/apache/thrift/pull/3547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "kpumuk (via GitHub)" <gi...@apache.org>.
kpumuk commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314514950
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against @max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
+ offset = 0
+ begin
+ while offset < compressed.bytesize
+ out = inflater.inflate(compressed.byteslice(offset, ZLIB_INFLATE_CHUNK_SIZE))
Review Comment:
There is a memory over-use here. I would recommend defining a `buffer = ''` variable outside of the loop, and the pass it here:
```suggestion
out = inflater.inflate(compressed.byteslice(offset, ZLIB_INFLATE_CHUNK_SIZE), buffer:)
```
Without it, uncompressing a large payload would require double memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bound decompressed size for Ruby HeaderTransport ZLIB transform [thrift]
Posted by "Jens-G (via GitHub)" <gi...@apache.org>.
Jens-G commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314558029
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against @max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
Review Comment:
Feel free to edit the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org