You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "stiga-huang (via GitHub)" <gi...@apache.org> on 2023/06/16 15:00:54 UTC
[GitHub] [thrift] stiga-huang opened a new pull request, #2818: THRIFT-5716: Fix uint32_t overflow in TMemoryBuffer
stiga-huang opened a new pull request, #2818:
URL: https://github.com/apache/thrift/pull/2818
<!-- Explain the changes in the pull request below: -->
This fixes the overflow in `TMemoryBuffer::ensureCanWrite()` when the current usage plus the length of new content exceeds the max of uint32_t (e.g. 4GB). The buggy code is
```cpp
const uint32_t current_used = bufferSize_ - avail;
const uint32_t required_buffer_size = len + current_used;
if (required_buffer_size > maxBufferSize_) {
throw TTransportException(...)
}
```
`len + current_used` could exceed the max value of `uint32_t`. We should define `current_used` as `uint64_t` so the result of the expression is also `uint64_t`. Changed the type of `required_buffer_size` to `uint64_t` as well.
Added a unit test for this.
<!-- We recommend you review the checklist/tips before submitting a pull request. -->
- [x] Did you create an [Apache Jira](https://issues.apache.org/jira/projects/THRIFT/issues/) ticket? ([Request account here](https://selfserve.apache.org/jira-account.html), not required for trivial changes)
- [x] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
- [x] Did you squash your changes to a single commit? (not required, but preferred)
- [x] Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
- [x] If your change does not involve any code, include `[skip ci]` anywhere in the commit message to free up build resources.
<!--
The Contributing Guide at:
https://github.com/apache/thrift/blob/master/CONTRIBUTING.md
has more details and tips for committing properly.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] stiga-huang commented on pull request #2818: THRIFT-5716: lib: cpp: Fix uint32_t overflow in TMemoryBuffer
Posted by "stiga-huang (via GitHub)" <gi...@apache.org>.
stiga-huang commented on PR #2818:
URL: https://github.com/apache/thrift/pull/2818#issuecomment-1595436982
Thanks for the quick review!
IIUC, `TMemoryBuffer` can be used to serialize any thrift object and the thrift object can exceed 4GB in the total size. E.g. in Impala, we have such a thrift object that contains `map<string, string>` and `list<string>`. The total length could be arbitrarily long:
https://github.com/apache/impala/blob/06eb62d3efa1c94810c4276f90896fa62205a49b/common/thrift/RuntimeProfile.thrift#L239-L242
We need to correctly reject such serialization, i.e. throw `TTransportException` about "Internal buffer size overflow". Currently, due to the overflow of `uint32_t`, `TMemoryBuffer::ensureCanWrite()` thought the write is legal and proceed without actually expanding the buffer (since `new_size` is smaller than the current size). It then writes to invalid memory and leads to a crash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] Jens-G merged pull request #2818: THRIFT-5716: lib: cpp: Fix crash in TMemoryBuffer due to uint32_t overflow
Posted by "Jens-G (via GitHub)" <gi...@apache.org>.
Jens-G merged PR #2818:
URL: https://github.com/apache/thrift/pull/2818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau commented on pull request #2818: THRIFT-5716: lib: cpp: Fix uint32_t overflow in TMemoryBuffer
Posted by "emmenlau (via GitHub)" <gi...@apache.org>.
emmenlau commented on PR #2818:
URL: https://github.com/apache/thrift/pull/2818#issuecomment-1594870591
Looks reasonable. But a naiive question: is thrift currently supporting any data size above 4GB?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy commented on pull request #2818: THRIFT-5716: lib: cpp: Fix uint32_t overflow in TMemoryBuffer
Posted by "fishy (via GitHub)" <gi...@apache.org>.
fishy commented on PR #2818:
URL: https://github.com/apache/thrift/pull/2818#issuecomment-1594888629
> Looks reasonable. But a naiive question: is thrift currently supporting any data size above 4GB?
From what I understand, on protocol level, all of [TBinaryProtocol](https://github.com/apache/thrift/blob/master/doc/specs/thrift-binary-protocol.md), [TCompactProtocol](https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md), and [THeaderProtocol](https://github.com/apache/thrift/blob/master/doc/specs/HeaderFormat.md) only support 32-bit lengths.
TFramedTransport does not have a spec (as far as I could tell) but the implementations should also all be limiting to 32-bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy commented on pull request #2818: THRIFT-5716: lib: cpp: Fix uint32_t overflow in TMemoryBuffer
Posted by "fishy (via GitHub)" <gi...@apache.org>.
fishy commented on PR #2818:
URL: https://github.com/apache/thrift/pull/2818#issuecomment-1594891779
Both TBinary and TCompact specs also explicitly stated that it's _signed_ 32-bit int. THeader spec doesn't explicitly call out signedness, but in general we are using signed ints across thrift because java's lack of support of unsigned ints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org