You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2026/04/26 02:48:45 UTC
[PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
mrhhsg opened a new pull request, #62843:
URL: https://github.com/apache/doris/pull/62843
### What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
`BitmapValue::deserialize(const char* src)` only accepted a pointer with no buffer length. The `SET_V2` branch read a `uint32_t` size from the payload and then `memcpy`d `size * 8` bytes without any bound check. Authenticated users could craft a base64 string passed to `bitmap_from_base64()` -> `BitmapValue::deserialize()` to trigger an out-of-bounds read of BE memory, crashing the process or potentially leaking heap contents. The same unsafe ctor is also reachable through `BitmapIntersect::deserialize` (the `orthogonal_bitmap_intersect*` aggregate states) and via PB / JSONB column-read paths.
This change:
- Tightens the public API: removes the unsafe single-pointer entries `BitmapValue(const char*)`, `BitmapValue::deserialize(const char*)`, `BitmapIntersect(const char*)` (both overloads) and `BitmapExprCalculation(const char*)`. The only deserialize signature is now `BitmapValue::deserialize(const char* src, size_t src_size, size_t* consumed_out = nullptr)`, which validates every header / payload read for `EMPTY`, `SINGLE32`, `SINGLE64`, `SET`, `SET_V2`, and `BITMAP*` (via a new `Roaring64Map::readSafe` that dispatches to croaring's bound-checked portable / native deserializer based on the V1/V2 header byte). Malformed inputs return `false` and reset the object to EMPTY so no failure path leaves a dangling `_bitmap` pointer or half-populated `_set`. NOTE: this is a public-header signature change; out-of-tree callers that still pass a bare `const char*` need to migrate to the length-aware overload.
- Adds a length-aware `BitmapIntersect::deserialize(src, src_size)` for both the templated and `std::string_view` specializations. It parses into a temporary map and only swaps into `_bitmaps` on full success, rejects duplicate keys via `try_emplace`, bound-checks variable-sized keys via new `Helper::read_from_safe` variants (`StringRef` / `std::string` / `VecDateTimeValue` / `DecimalV2Value`), and rejects trailing garbage.
- Hardens the `Block` -> `IDataType::deserialize` chain: plumbs a trusted `buf_end` through every `IDataType::deserialize` impl. For complex types (array/map/varbinary/quantilestate/fixed_length_object/string/number/HLL) reorders check-then-resize and adds `n > SIZE_MAX/elem_size` overflow guards before computing payload byte counts. `Block::deserialize` no longer trusts the pre-allocated scratch length on the compressed path: it returns `Status::Corruption` when the codec produces fewer bytes than the metadata claims (was a debug-only `DCHECK`), so bound checks downstream cannot mistake uninitialized scratch tail for input. Snappy fallback also promotes `DCHECK` to release-checked `Status::Corruption`.
- Hardens `SingleValueDataComplexType::read()`: reads the length as `uint64_t` and validates it against the new `BufferReadable::remaining()` before constructing a `buf_end`, preventing a corrupted aggregate-state payload from forging a fake boundary.
- Updates attacker-reachable / known-length callers to pass the actual payload length, enforce strict consumption (`getSizeInBytes() == src_size`), and fail loudly (return error / throw `Exception(INTERNAL_ERROR)`) instead of silently corrupting data: `bitmap_from_base64`, `DataTypeBitMapSerDe::deserialize_one_cell_from_json` / `from_olap_string` / `read_column_from_pb` / `read_one_cell_from_jsonb`, `DataTypeBitMap::deserialize(_as_stream)`, `ColumnComplexType<TYPE_BITMAP>::insert_binary_data`, and `AggOrthBitmapBaseDataCount::read`.
- Hardens `BufferReadable` so every read is bounded by the underlying `StringRef`: `read(size_t)`, `read(char*, size_t)`, `read_binary(Type&)`, `read_binary(String&)` / `PaddedPODArray<UInt8>&`, `read_binary_into`, and `add_offset` all throw `Exception(INTERNAL_ERROR)` if the requested length exceeds `remaining()`. `read_var_uint` now also rejects oversized length prefixes (>9 bytes) before the inner `read()` call, so a forged byte cannot be used to drive bulk allocations.
- Wraps the third-party `streamvbyte_decode` (used by the column encoders for string offsets / nullable maps / number / decimal / fixed-length-object payloads) with `streamvbyte_decode_safe`: it copies the encoded slice into a STREAMVBYTE_PADDING-padded scratch buffer (the upstream decoder may read up to 16 trailing bytes), and verifies the decoder consumed exactly `encode_size` bytes. A mismatch is reported as `Status::Corruption` so a crafted column body cannot under/over-read the payload.
- Enforces the `ColumnConst` invariant in `deserialize_const_flag_and_row_num`: when `is_const_column` is set, the serialized `real_have_saved_num` must be `1`. Crafted headers that violate this are rejected with `Exception(INTERNAL_ERROR)` before reaching `ColumnConst::create`.
- Adds BE unit tests covering oversized SET_V2 size, truncated SET_V2 header, truncated SET, truncated SINGLE32/64, empty buffer, truncated BITMAP* payload (target stays empty / safe to reuse), trailing-garbage rejection (standalone bitmap and BitmapIntersect), V2 round-trip, duplicate-key rejection in `BitmapIntersect`, and partial-state rejection in `BitmapIntersect`. Adds `BufferReadable` bound-check tests (out-of-range `read`, truncated and oversized varint prefix, oversized string size header) and `streamvbyte_decode_safe` round-trip and size-mismatch tests.
### Release note
Fix BE crash / heap memory disclosure when deserializing a malformed bitmap payload (e.g. via `bitmap_from_base64` or
`orthogonal_bitmap_intersect*` aggregate state).
### Check List (For Author)
- Test:
- Unit Test (be/test/core/value/bitmap_value_test.cpp)
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@doris.apache.org>.
github-actions[bot] closed pull request #62843: [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read
URL: https://github.com/apache/doris/pull/62843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62843:
URL: https://github.com/apache/doris/pull/62843#discussion_r3145426279
##########
be/src/core/value/bitmap_value.h:
##########
@@ -1936,13 +2002,31 @@ class BitmapValue {
}
// Deserialize a bitmap value from `src`.
- // Return false if `src` begins with unknown type code, true otherwise.
+ // Return false if `src` begins with unknown type code or the bounded input is truncated.
Review Comment:
This still keeps an unbounded entry point alive, and `BitmapIntersect::deserialize()` in `be/src/util/bitmap_intersect.h` is still using it (`BitmapValue(reader)` with no length). That means the `intersect_count` / `orthogonal_bitmap_intersect*` aggregate-state path is not actually hardened yet: a truncated state buffer can still walk past the end of the `BufferReadable` payload.
##########
be/src/core/data_type/data_type_bitmap.cpp:
##########
@@ -90,8 +90,9 @@ const char* DataTypeBitMap::deserialize(const char* buf, MutableColumnPtr* colum
const auto* meta_ptr = reinterpret_cast<const size_t*>(buf);
const char* data_ptr = buf + sizeof(size_t) * real_have_saved_num;
for (size_t i = 0; i < real_have_saved_num; ++i) {
- data[i].deserialize(data_ptr);
- data_ptr += unaligned_load<size_t>(&meta_ptr[i]);
+ size_t bitmap_size = unaligned_load<size_t>(&meta_ptr[i]);
+ data[i].deserialize(data_ptr, bitmap_size);
Review Comment:
These bounded calls now have a failure signal, but it is ignored here (and in `deserialize_as_stream()` below). On malformed BITMAP payloads `_deserialize()` already mutates the target before it returns `false` (`_type = BITMAP` is assigned before `readSafe()` succeeds), so block / stream deserialization can continue with a stale or invalid `BitmapValue` instead of rejecting corruption at this boundary.
##########
be/src/core/data_type_serde/data_type_bitmap_serde.cpp:
##########
@@ -98,7 +98,10 @@ Status DataTypeBitMapSerDe::write_column_to_pb(const IColumn& column, PValues& r
Status DataTypeBitMapSerDe::read_column_from_pb(IColumn& column, const PValues& arg) const {
auto& col = reinterpret_cast<ColumnBitmap&>(column);
for (int i = 0; i < arg.bytes_value_size(); ++i) {
- BitmapValue value(arg.bytes_value(i).data());
+ BitmapValue value;
+ if (!value.deserialize(Slice(arg.bytes_value(i).data(), arg.bytes_value(i).size()))) {
Review Comment:
This fixes the PB caller, but the JSONB path in the same file still bypasses it: `read_one_cell_from_jsonb()` below still does `BitmapValue(blob->getBlob())` with no length. A truncated or corrupt JSONB binary cell will still hit the old unbounded parser and can read past the blob boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #62843:
URL: https://github.com/apache/doris/pull/62843#issuecomment-4328047504
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62843:
URL: https://github.com/apache/doris/pull/62843#issuecomment-4321115665
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #62843:
URL: https://github.com/apache/doris/pull/62843#issuecomment-4324779567
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound-check BitmapValue::deserialize SET_V2 payload to fix OOB read [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62843:
URL: https://github.com/apache/doris/pull/62843#issuecomment-6391347813
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org