You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lucasfang (via GitHub)" <gi...@apache.org> on 2026/09/14 07:58:04 UTC
[PR] perf(read): skip the payload Concatenate when late materialization matches a single run [paimon-cpp]
lucasfang opened a new pull request, #336:
URL: https://github.com/apache/paimon-cpp/pull/336
<!-- PR title: perf(read): skip the payload Concatenate when late materialization matches a single run -->
### Purpose
Linked issue: close #335
The payload pass of late materialization compacts every batch down to the rows that passed the probe pass: `LateMaterializingFileBatchReader::ReadPayloadBatch` turns the matched bitmap into one zero-copy slice per contiguous run (`ReaderUtils::GenerateFilteredArrayVector`), then merges the runs with `arrow::Concatenate`. When the matched rows form a single run there is nothing to merge, so `Concatenate` only copies the payload bytes a second time to move the slice back to offset zero — and that rebase is redundant, because `AssembleFullBatch` already passes every column it takes through `ArrowUtils::NormalizeArrayOffsets`. A single run is the common shape here: the payload pass re-reads with the matched rows as its selection, so a precisely filtering reader returns fully matched batches, and a range predicate over clustered data keeps whole batches at a time.
This PR hands a lone slice straight to `AssembleFullBatch` and keeps `Concatenate` for two runs or more. Neither half of the single-run case is worse than before: a run starting at the batch head reaches the output with no copy at all (`NeedsNormalization` is false, so normalization is a no-op), and a run starting mid-batch is still copied exactly once, by the per-column normalization instead of by `Concatenate`. The output is unchanged — the assembled batch is still built from normalized, equal-length columns, so the exported `ArrowArray` keeps the offset-zero, children-match-parent shape consumers rely on.
Known trade-off: where the run stays zero-copy it keeps the inner batch's buffers alive until the consumer releases the assembled batch. That retains more than the matched rows only when the run starts at the head without reaching the end of the batch, and stays bounded by the read batch size times the prefetch queue depth.
### Tests
New `LateMaterializingFileBatchReaderTest.DictionaryPayloadColumnKeepsEncoding`: a dictionary-encoded payload column (`dictionary<int32, utf8>`, built with `StringDictionary32Builder` so the indices match the declared type) read under `k >= 5` with a fixed batch size, which reaches both shapes of a lone run — one at batch offset 0 that passes through untouched, one at batch offset 2 that the per-column normalization copies through `CopyToZeroOffset`, the one type whose normalization is a full copy rather than a buffer rebase. The test asserts the output stays dictionary-encoded and decodes to the expected values. Row-level correctness of the single-run and multi-run paths is already covered by the existing `ContiguousSubsetAcrossBatches`, `ScatteredAlternatingMatch`, `NestedPayloadColumn`, and `WorksAsInnerOfPrefetchReader` cases.
Verified in a Debug build (`-j 96`):
- `paimon-common-test --gtest_filter='LateMaterializingFileBatchReaderTest.*' --gtest_repeat=20`: 17 tests passed on every repetition (the mock reader randomizes batch sizes elsewhere in the suite, so repetition covers varying run shapes).
- `paimon-common-test`: 1620 of 1622 tests from 185 suites passed, 2 skipped (pre-existing skips in `prefetch_file_batch_reader_impl_test.cpp`).
- `paimon-core-test`: 2073 tests from 233 suites passed.
- `pre-commit run --files <the two changed files>` and `git diff --check`: clean.
### API and Format
No. No header under `include/paimon/` is touched, and neither the storage format nor the protocol changes. The reader's externally observable output is byte-for-byte the same batch.
### Documentation
No. This is an internal read-path optimization with no user-visible API, configuration, or behavior change.
### Generative AI tooling
Generated-by: Qoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(read): skip the payload Concatenate when late materialization matches a single run [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 commented on code in PR #336:
URL: https://github.com/apache/paimon-cpp/pull/336#discussion_r4004291992
##########
src/paimon/common/utils/arrow/arrow_utils.cpp:
##########
@@ -432,6 +464,56 @@ uint64_t ArrowUtils::GetArrayMemoryUsage(const std::shared_ptr<arrow::ArrayData>
return result;
}
+std::vector<char> ArrowUtils::UnpackBooleansToBytes(const arrow::BooleanArray& array, bool negate) {
+ const int64_t length = array.length();
+ std::vector<char> is_valid(static_cast<size_t>(length));
+ if (length == 0) {
+ return is_valid;
+ }
+
+ const std::shared_ptr<arrow::ArrayData>& data = array.data();
+ // `BooleanArray::Value(i)` and `Array::IsValid(i)` both read their bitmap at bit `i + offset`
+ // from the start of the buffer, so one bit index serves the value and the validity alike.
+ const int64_t bit_offset = data->offset;
+ const uint8_t* values = data->GetValuesSafe<uint8_t>(/*i=*/1, /*absolute_offset=*/0);
+ const uint8_t* validity = array.null_bitmap_data();
+ // Without a validity bitmap `Array::IsValid` falls back to `null_count != length`, which makes
+ // every row valid unless the array is null throughout. One that is has no row left to unpack,
+ // and is the only array whose value bitmap is not read.
+ if (validity == nullptr && data->null_count == length) {
+ return is_valid;
+ }
+
+ auto unpack_row = [&](int64_t row) -> char {
+ const int64_t bit = bit_offset + row;
+ const bool holds_value = validity == nullptr || arrow::bit_util::GetBit(validity, bit);
+ return static_cast<char>(holds_value && (arrow::bit_util::GetBit(values, bit) != negate));
+ };
+
+ char* out = is_valid.data();
+ int64_t row = 0;
+ // The rows sharing the leading partial byte, which the byte at a time body cannot take whole.
+ // None when the bit offset is already on a byte boundary, as it is for the batch a kernel has
+ // just written.
+ const int64_t head = std::min<int64_t>(length, (8 - (bit_offset & 7)) & 7);
+ for (; row < head; row++) {
+ out[row] = unpack_row(row);
+ }
+ for (; row + 8 <= length; row += 8) {
+ const int64_t byte = (bit_offset + row) >> 3;
+ // Every row holds a value when there is no validity bitmap, so all eight bits pass.
+ const uint8_t valid_byte = validity == nullptr ? uint8_t{0xFF} : validity[byte];
+ const uint8_t value_byte = values[byte];
+ const uint8_t bits = negate ? static_cast<uint8_t>(~value_byte & valid_byte)
+ : static_cast<uint8_t>(value_byte & valid_byte);
+ std::memcpy(out + row, kExpandedBytes[bits].bytes, sizeof(ExpandedByte));
+ }
+ for (; row < length; row++) {
+ out[row] = unpack_row(row);
+ }
+ return is_valid;
Review Comment:
Consider use `arrow::util::bit_util::bits_to_bytes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(read): skip the payload Concatenate when late materialization matches a single run [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 merged PR #336:
URL: https://github.com/apache/paimon-cpp/pull/336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(read): skip the payload Concatenate when late materialization matches a single run [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 commented on code in PR #336:
URL: https://github.com/apache/paimon-cpp/pull/336#discussion_r4011630718
##########
src/paimon/common/utils/arrow/arrow_utils_test.cpp:
##########
@@ -1215,4 +1217,124 @@ TEST(ArrowUtilsTest, TestFlattenUnresolvableDictionaries) {
}
}
+namespace {
+
+/// What `ArrowUtils::UnpackBooleansToBytes` has to agree with: every row taken one at a time
+/// through the very accessors the helper replaces.
+std::vector<char> UnpackRowByRow(const arrow::BooleanArray& array, bool negate) {
+ std::vector<char> is_valid(static_cast<size_t>(array.length()), 0);
+ for (int64_t i = 0; i < array.length(); i++) {
+ if (array.IsNull(i)) {
+ continue;
+ }
+ is_valid[i] = static_cast<char>(array.Value(i) != negate);
+ }
+ return is_valid;
+}
+
+/// A boolean array of `values.size()` rows holding the value `values[i]` at row `i`, with a
+/// validity bitmap that marks row `i` as holding one when `valid[i]` is true.
+std::shared_ptr<arrow::BooleanArray> BooleansOf(const std::vector<bool>& values,
+ const std::vector<bool>& valid) {
+ const auto length = static_cast<int64_t>(values.size());
+ std::shared_ptr<arrow::Buffer> value_bits = arrow::AllocateBitmap(length).ValueOrDie();
+ std::shared_ptr<arrow::Buffer> valid_bits = arrow::AllocateBitmap(length).ValueOrDie();
+ int64_t null_count = 0;
+ for (int64_t i = 0; i < length; i++) {
+ arrow::bit_util::SetBitTo(value_bits->mutable_data(), i, values[static_cast<size_t>(i)]);
+ arrow::bit_util::SetBitTo(valid_bits->mutable_data(), i, valid[static_cast<size_t>(i)]);
+ null_count += valid[static_cast<size_t>(i)] ? 0 : 1;
+ }
+ return std::make_shared<arrow::BooleanArray>(length, value_bits, valid_bits, null_count);
+}
+
+} // namespace
+
+TEST(ArrowUtilsTest, TestUnpackBooleansToBytes) {
+ std::shared_ptr<arrow::Array> made =
+ arrow::ipc::internal::json::ArrayFromJSON(arrow::boolean(), R"([true, false, null, true])")
+ .ValueOrDie();
+ const auto& array = checked_cast<const arrow::BooleanArray&>(*made);
+
+ // A null row is left at 0 whatever the value bitmap holds for it.
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(array, /*negate=*/false),
+ std::vector<char>({1, 0, 0, 1}));
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(array, /*negate=*/true),
+ std::vector<char>({0, 1, 0, 0}));
+}
+
+TEST(ArrowUtilsTest, TestUnpackBooleansToBytesAgreesWithTheAccessorsOnEverySlice) {
+ // Neither the row count nor the periods the values and the nulls repeat on are a multiple of
+ // eight, so the slices below land on a partial leading byte, on whole bytes and on a partial
+ // trailing one, whichever offset and length they take.
+ constexpr int64_t kRows = 41;
+ std::vector<bool> values;
+ std::vector<bool> valid;
+ for (int64_t i = 0; i < kRows; i++) {
+ values.push_back((i * 7 + 3) % 5 != 0);
+ valid.push_back(i % 4 != 0);
+ }
+ const std::shared_ptr<arrow::BooleanArray> source = BooleansOf(values, valid);
+
+ for (int64_t offset = 0; offset <= kRows; offset++) {
+ for (int64_t length = 0; length <= kRows - offset; length++) {
+ // `Slice` returns a new owner, so it has to be held for as long as the slice is read.
+ std::shared_ptr<arrow::Array> sliced = source->Slice(offset, length);
+ const auto& slice = checked_cast<const arrow::BooleanArray&>(*sliced);
+ for (bool negate : {false, true}) {
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(slice, negate),
+ UnpackRowByRow(slice, negate))
+ << "offset=" << offset << " length=" << length << " negate=" << negate;
+ }
+ }
+ }
+}
+
+TEST(ArrowUtilsTest, TestUnpackBooleansToBytesWithoutAValidityBitmap) {
+ // A bitmap written for a column with no null in it carries no validity buffer at all, which
+ // `Array::IsValid` reads as every row holding a value.
+ const int64_t length = 10;
+ std::shared_ptr<arrow::Buffer> value_bits = arrow::AllocateBitmap(length).ValueOrDie();
+ for (int64_t i = 0; i < length; i++) {
+ arrow::bit_util::SetBitTo(value_bits->mutable_data(), i, i % 3 != 0);
+ }
+ const arrow::BooleanArray array(length, value_bits, /*null_bitmap=*/nullptr,
+ /*null_count=*/0);
+
+ std::vector<char> expected(static_cast<size_t>(length));
+ for (int64_t i = 0; i < length; i++) {
+ expected[static_cast<size_t>(i)] = static_cast<char>(i % 3 != 0);
+ }
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(array, /*negate=*/false), expected);
+ ASSERT_EQ(UnpackRowByRow(array, /*negate=*/false), expected);
+
+ // `negate` still applies once every row holds a value.
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(array, /*negate=*/true),
+ UnpackRowByRow(array, /*negate=*/true));
+}
+
+TEST(ArrowUtilsTest, TestUnpackBooleansToBytesOfAnArrayThatIsNullThroughout) {
+ // Without a validity bitmap `Array::IsValid` falls back to `null_count != length`, so an array
+ // whose null count is its length holds no value at any row, however its value bitmap reads.
+ const int64_t length = 9;
+ std::shared_ptr<arrow::Buffer> value_bits = arrow::AllocateBitmap(length).ValueOrDie();
+ arrow::bit_util::SetBitsTo(value_bits->mutable_data(), /*offset=*/0, length, /*bits=*/true);
+ const arrow::BooleanArray array(length, value_bits, /*null_bitmap=*/nullptr,
+ /*null_count=*/length);
+
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(array, /*negate=*/false),
+ std::vector<char>(static_cast<size_t>(length), 0));
+ ASSERT_EQ(ArrowUtils::UnpackBooleansToBytes(array, /*negate=*/true),
+ std::vector<char>(static_cast<size_t>(length), 0));
+ ASSERT_EQ(UnpackRowByRow(array, /*negate=*/true),
+ std::vector<char>(static_cast<size_t>(length), 0));
Review Comment:
Isn’t this an invalid array? Could you confirm that arrow `Validate` should already fail for this case (see `arrow/array/validate.cc`)? If so, testing an invalid array here does not seem very meaningful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org