You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "kecookier (via GitHub)" <gi...@apache.org> on 2026/09/09 11:22:42 UTC
[PR] Bugfix/fix rss sort reader eos full [gluten]
kecookier opened a new pull request, #12984:
URL: https://github.com/apache/gluten/pull/12984
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
### Background
Two bugs were discovered in our internal production environment: shuffle read tasks hit an infinite loop or deserialization failures with Celeborn (rss_sort + hash partitioning).
### What
1. EOS mid-page infinite loop. GlutenByteInputStream::readBytes drives next(true) in a for(;;) loop, but VeloxInputStream::next() ignored the throwIfPastEnd argument and silently returned on EOS — a truncated partition stream made the loop spin forever (task stuck in RUNNABLE). The Read() result was also stored into an unsigned offset_ without a signed guard, so a negative return corrupted setRange.
Fix: next() honors throwIfPastEnd (fails fast on EOS when bytes are still expected), guards the signed Read() result, and the constructor / hasNext() probe with next(false) so a clean EOS on an empty stream stays graceful.
2. Multi-window page corruption. VeloxInputStream is a single-window refill stream — each next() overwrites the sole ByteRange — but the Presto serde assumes stable multi-range data (tellp → pre-scan → seekp back for nested types, nextView over the payload for checksum verification). When a page spans multiple read windows, the serde's backward seek read window data already overwritten by a refill, corrupting the stream (e.g. Invalid serialized string size).
Fix: VeloxInputStream overrides nextView (refills the window instead of reporting EOS at a range boundary) and tellp/seekp (window-bounded, fail fast). The deserializer gained readPage(): fast path deserializes in-situ when the whole page fits in the current window; slow path reassembles cross-window pages (copied header stitched with the in-situ payload, or a fully copied payload) into a contiguous BufferInputStream so serde seeks never touch refill-overwritten data.
The Presto page wire header is parsed manually (21 bytes, frozen wire format) to avoid depending on velox-internal serde detail headers.
Regression tests in VeloxShuffleReaderTest cover both fixes, driven through the public VeloxRssSortShuffleReaderDeserializer API.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "kecookier (via GitHub)" <gi...@apache.org>.
kecookier commented on code in PR #12984:
URL: https://github.com/apache/gluten/pull/12984#discussion_r4047004537
##########
cpp/velox/shuffle/VeloxShuffleReader.cc:
##########
@@ -948,13 +1011,173 @@ void VeloxRssSortShuffleReaderDeserializer::loadNextStream() {
constexpr uint64_t kMaxReadBufferSize = (1 << 20) - AlignedBuffer::kPaddedSize;
auto buffer = AlignedBuffer::allocate<char>(kMaxReadBufferSize, memoryManager_->getLeafMemoryPool().get());
- in_ = std::make_unique<VeloxInputStream>(std::move(arrowIn_), std::move(buffer));
+ in_ = std::make_unique<RssSortShuffleReaderInputStream>(std::move(arrowIn_), std::move(buffer));
}
-size_t VeloxRssSortShuffleReaderDeserializer::VeloxInputStream::remainingSize() const {
+size_t VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::remainingSize() const {
return std::numeric_limits<unsigned long>::max();
}
+uint8_t VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::readByte() {
+ if (current_->position < current_->size) {
+ return current_->buffer[current_->position++];
+ }
+ next();
+ return readByte();
+}
+
+void VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::readBytes(uint8_t* bytes, int32_t size) {
+ VELOX_CHECK_GE(size, 0, "Attempting to read negative number of bytes");
+ int32_t offset = 0;
+ for (;;) {
+ int32_t available = current_->size - current_->position;
+ int32_t numUsed = std::min(available, size);
+ simd::memcpy(bytes + offset, current_->buffer + current_->position, numUsed);
+ offset += numUsed;
+ size -= numUsed;
+ current_->position += numUsed;
+ if (!size) {
+ return;
+ }
+ next();
+ }
+}
+
+void VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::skip(int32_t size) {
+ VELOX_CHECK_GE(size, 0, "Attempting to skip negative number of bytes");
+ for (;;) {
+ int32_t available = current_->size - current_->position;
+ int32_t numUsed = std::min(available, size);
+ size -= numUsed;
+ current_->position += numUsed;
+ if (!size) {
+ return;
+ }
+ next();
+ }
+}
+
+std::string VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::toString() const {
+ std::stringstream oss;
+ oss << ranges_.size() << " ranges (position/size) [";
+ for (const auto& range : ranges_) {
+ oss << "(" << range.position << "/" << range.size << (&range == current_ ? " current" : "") << ")";
+ if (&range != &ranges_.back()) {
+ oss << ",";
+ }
+ }
+ oss << "]";
+ return oss.str();
+}
+
+std::string_view VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::nextView(int64_t size) {
+ VELOX_CHECK_GE(size, 0, "Attempting to view negative number of bytes");
+ if (ranges_.empty()) {
+ return std::string_view(nullptr, 0);
+ }
+ if (ranges_[0].position == ranges_[0].size) {
+ // Current window is exhausted. For single-window refill streams, next()
+ // overwrites the window with fresh data, so attempt refill before
+ // reporting end-of-stream.
+ next(false);
+ if (ranges_.empty() || ranges_[0].position == ranges_[0].size) {
+ return std::string_view(nullptr, 0);
+ }
+ }
+ VELOX_DCHECK(ranges_[0].size > 0);
+ const int32_t position = ranges_[0].position;
+ const int64_t viewSize = std::min<int64_t>(ranges_[0].size - ranges_[0].position, size);
+ ranges_[0].position += static_cast<int32_t>(viewSize);
+ return std::string_view(reinterpret_cast<char*>(ranges_[0].buffer) + position, viewSize);
+}
+
+std::streampos VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::tellp() const {
+ if (ranges_.empty()) {
+ return 0;
+ }
+ return static_cast<std::streampos>(static_cast<int64_t>(totalBytesRead_) - (ranges_[0].size - ranges_[0].position));
+}
+
+void VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::seekp(std::streampos position) {
+ if (ranges_.empty() && position == 0) {
+ return;
+ }
+ VELOX_CHECK(!ranges_.empty(), "Cannot seek an empty RssSortShuffleReaderInputStream");
+ const int64_t windowStart = static_cast<int64_t>(totalBytesRead_) - ranges_[0].size;
+ const int64_t windowEnd = static_cast<int64_t>(totalBytesRead_);
+ const int64_t target = static_cast<int64_t>(position);
+ VELOX_CHECK(
+ target >= windowStart && target <= windowEnd,
+ "RssSortShuffleReaderInputStream::seekp({}) is outside the resident window [{}, {}): bytes before the "
+ "window were already consumed from the underlying stream (totalBytesRead={})",
+ target,
+ windowStart,
+ windowEnd,
+ totalBytesRead_);
+ ranges_[0].position = static_cast<int32_t>(target - windowStart);
+}
+
+RowVectorPtr VeloxRssSortShuffleReaderDeserializer::readPage() {
+ using facebook::velox::serializer::presto::detail::kCompressedBitMask;
+ using facebook::velox::serializer::presto::detail::kHeaderSize;
+ using facebook::velox::serializer::presto::detail::PrestoHeader;
+ constexpr int32_t kPrestoHeaderSize = kHeaderSize;
+
+ // Fast path: peek the header without consuming; if the whole page fits in
+ // the current window, deserialize in-situ from in_ directly.
+ if (in_->remainingInWindow() >= kPrestoHeaderSize) {
+ std::string_view window(reinterpret_cast<const char*>(in_->data()), in_->remainingInWindow());
+ auto peekedHeader = PrestoHeader::read(&window);
+ if (peekedHeader.has_value()) {
+ const int32_t payloadSize = (peekedHeader->pageCodecMarker & kCompressedBitMask) != 0
+ ? peekedHeader->compressedSize
+ : peekedHeader->uncompressedSize;
+ const int64_t totalSize = kPrestoHeaderSize + static_cast<int64_t>(payloadSize);
+ if (totalSize <= in_->remainingInWindow()) {
+ RowVectorPtr rowVector;
+ VectorStreamGroup::read(
+ in_.get(), memoryManager_->getLeafMemoryPool().get(), rowType_, serde_, &rowVector, &serdeOptions_);
+ return rowVector;
+ }
+ }
+ }
+
+ // Slow path: the page spans multiple read windows. Reassemble it into a
+ // contiguous BufferInputStream so the serde's backward seek never touches
+ // window data overwritten by a refill.
+ std::array<uint8_t, kHeaderSize> headerStorage;
+ in_->readBytes(headerStorage.data(), kHeaderSize);
+ std::string_view headerBytes(reinterpret_cast<const char*>(headerStorage.data()), kHeaderSize);
+ auto headerOpt = PrestoHeader::read(&headerBytes);
+ VELOX_CHECK(headerOpt.has_value(), "Invalid Presto page header");
+ const auto& header = *headerOpt;
+
+ const int32_t payloadSize =
+ (header.pageCodecMarker & kCompressedBitMask) != 0 ? header.compressedSize : header.uncompressedSize;
Review Comment:
Not needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12984:
URL: https://github.com/apache/gluten/pull/12984#discussion_r4046962275
##########
cpp/velox/shuffle/VeloxShuffleReader.cc:
##########
@@ -948,13 +1011,173 @@ void VeloxRssSortShuffleReaderDeserializer::loadNextStream() {
constexpr uint64_t kMaxReadBufferSize = (1 << 20) - AlignedBuffer::kPaddedSize;
auto buffer = AlignedBuffer::allocate<char>(kMaxReadBufferSize, memoryManager_->getLeafMemoryPool().get());
- in_ = std::make_unique<VeloxInputStream>(std::move(arrowIn_), std::move(buffer));
+ in_ = std::make_unique<RssSortShuffleReaderInputStream>(std::move(arrowIn_), std::move(buffer));
}
-size_t VeloxRssSortShuffleReaderDeserializer::VeloxInputStream::remainingSize() const {
+size_t VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::remainingSize() const {
return std::numeric_limits<unsigned long>::max();
}
+uint8_t VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::readByte() {
+ if (current_->position < current_->size) {
+ return current_->buffer[current_->position++];
+ }
+ next();
+ return readByte();
+}
+
+void VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::readBytes(uint8_t* bytes, int32_t size) {
+ VELOX_CHECK_GE(size, 0, "Attempting to read negative number of bytes");
+ int32_t offset = 0;
+ for (;;) {
+ int32_t available = current_->size - current_->position;
+ int32_t numUsed = std::min(available, size);
+ simd::memcpy(bytes + offset, current_->buffer + current_->position, numUsed);
+ offset += numUsed;
+ size -= numUsed;
+ current_->position += numUsed;
+ if (!size) {
+ return;
+ }
+ next();
+ }
+}
+
+void VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::skip(int32_t size) {
+ VELOX_CHECK_GE(size, 0, "Attempting to skip negative number of bytes");
+ for (;;) {
+ int32_t available = current_->size - current_->position;
+ int32_t numUsed = std::min(available, size);
+ size -= numUsed;
+ current_->position += numUsed;
+ if (!size) {
+ return;
+ }
+ next();
+ }
+}
+
+std::string VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::toString() const {
+ std::stringstream oss;
+ oss << ranges_.size() << " ranges (position/size) [";
+ for (const auto& range : ranges_) {
+ oss << "(" << range.position << "/" << range.size << (&range == current_ ? " current" : "") << ")";
+ if (&range != &ranges_.back()) {
+ oss << ",";
+ }
+ }
+ oss << "]";
+ return oss.str();
+}
+
+std::string_view VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::nextView(int64_t size) {
+ VELOX_CHECK_GE(size, 0, "Attempting to view negative number of bytes");
+ if (ranges_.empty()) {
+ return std::string_view(nullptr, 0);
+ }
+ if (ranges_[0].position == ranges_[0].size) {
+ // Current window is exhausted. For single-window refill streams, next()
+ // overwrites the window with fresh data, so attempt refill before
+ // reporting end-of-stream.
+ next(false);
+ if (ranges_.empty() || ranges_[0].position == ranges_[0].size) {
+ return std::string_view(nullptr, 0);
+ }
+ }
+ VELOX_DCHECK(ranges_[0].size > 0);
+ const int32_t position = ranges_[0].position;
+ const int64_t viewSize = std::min<int64_t>(ranges_[0].size - ranges_[0].position, size);
+ ranges_[0].position += static_cast<int32_t>(viewSize);
+ return std::string_view(reinterpret_cast<char*>(ranges_[0].buffer) + position, viewSize);
+}
+
+std::streampos VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::tellp() const {
+ if (ranges_.empty()) {
+ return 0;
+ }
+ return static_cast<std::streampos>(static_cast<int64_t>(totalBytesRead_) - (ranges_[0].size - ranges_[0].position));
+}
+
+void VeloxRssSortShuffleReaderDeserializer::RssSortShuffleReaderInputStream::seekp(std::streampos position) {
+ if (ranges_.empty() && position == 0) {
+ return;
+ }
+ VELOX_CHECK(!ranges_.empty(), "Cannot seek an empty RssSortShuffleReaderInputStream");
+ const int64_t windowStart = static_cast<int64_t>(totalBytesRead_) - ranges_[0].size;
+ const int64_t windowEnd = static_cast<int64_t>(totalBytesRead_);
+ const int64_t target = static_cast<int64_t>(position);
+ VELOX_CHECK(
+ target >= windowStart && target <= windowEnd,
+ "RssSortShuffleReaderInputStream::seekp({}) is outside the resident window [{}, {}): bytes before the "
+ "window were already consumed from the underlying stream (totalBytesRead={})",
+ target,
+ windowStart,
+ windowEnd,
+ totalBytesRead_);
+ ranges_[0].position = static_cast<int32_t>(target - windowStart);
+}
+
+RowVectorPtr VeloxRssSortShuffleReaderDeserializer::readPage() {
+ using facebook::velox::serializer::presto::detail::kCompressedBitMask;
+ using facebook::velox::serializer::presto::detail::kHeaderSize;
+ using facebook::velox::serializer::presto::detail::PrestoHeader;
+ constexpr int32_t kPrestoHeaderSize = kHeaderSize;
+
+ // Fast path: peek the header without consuming; if the whole page fits in
+ // the current window, deserialize in-situ from in_ directly.
+ if (in_->remainingInWindow() >= kPrestoHeaderSize) {
+ std::string_view window(reinterpret_cast<const char*>(in_->data()), in_->remainingInWindow());
+ auto peekedHeader = PrestoHeader::read(&window);
+ if (peekedHeader.has_value()) {
+ const int32_t payloadSize = (peekedHeader->pageCodecMarker & kCompressedBitMask) != 0
+ ? peekedHeader->compressedSize
+ : peekedHeader->uncompressedSize;
+ const int64_t totalSize = kPrestoHeaderSize + static_cast<int64_t>(payloadSize);
+ if (totalSize <= in_->remainingInWindow()) {
+ RowVectorPtr rowVector;
+ VectorStreamGroup::read(
+ in_.get(), memoryManager_->getLeafMemoryPool().get(), rowType_, serde_, &rowVector, &serdeOptions_);
+ return rowVector;
+ }
+ }
+ }
+
+ // Slow path: the page spans multiple read windows. Reassemble it into a
+ // contiguous BufferInputStream so the serde's backward seek never touches
+ // window data overwritten by a refill.
+ std::array<uint8_t, kHeaderSize> headerStorage;
+ in_->readBytes(headerStorage.data(), kHeaderSize);
+ std::string_view headerBytes(reinterpret_cast<const char*>(headerStorage.data()), kHeaderSize);
+ auto headerOpt = PrestoHeader::read(&headerBytes);
+ VELOX_CHECK(headerOpt.has_value(), "Invalid Presto page header");
+ const auto& header = *headerOpt;
+
+ const int32_t payloadSize =
+ (header.pageCodecMarker & kCompressedBitMask) != 0 ? header.compressedSize : header.uncompressedSize;
Review Comment:
The page-size fields are signed `int32_t`, but this path accepts a negative value as valid. For a corrupt uncompressed header, `payloadSize <= remainingInWindow()` succeeds, then `advance(payloadSize)` moves the stream position backwards and `data() - payloadSize` forms an invalid payload range instead of rejecting the page. Validate the selected payload size as non-negative before using it for pointer arithmetic or allocation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12984:
URL: https://github.com/apache/gluten/pull/12984#discussion_r4044281090
##########
cpp/velox/shuffle/VeloxShuffleReader.cc:
##########
@@ -865,10 +911,18 @@ void VeloxRssSortShuffleReaderDeserializer::VeloxInputStream::next(bool throwIfP
GLUTEN_ASSIGN_OR_THROW(int64_t realBytes, in_->Read(readBytes, buffer_->asMutable<char>()));
if (realBytes > 0) {
offset_ = realBytes;
+ totalBytesRead_ += realBytes;
+ atEnd_ = false;
setRange({buffer_->asMutable<uint8_t>(), static_cast<int32_t>(realBytes), 0});
- } else if (throwIfPastEnd) {
- VELOX_FAIL(
- "Reading past end of VeloxRssSortShuffleReaderDeserializer::VeloxInputStream, real bytes = {}", realBytes);
+ } else {
Review Comment:
`Read()` uses zero to signal EOS, but this branch also treats any negative `realBytes` as a clean end-of-stream. A buggy upstream can therefore silently truncate the partition and return no batch instead of failing fast (the negative-return case is part of #12985). Validate that the result is non-negative before setting `atEnd_`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma merged PR #12984:
URL: https://github.com/apache/gluten/pull/12984
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #12984:
URL: https://github.com/apache/gluten/pull/12984#issuecomment-5728601582
@kecookier `GlutenByteInputStream` is a copy from Velox code because they removed the original one: https://github.com/apache/gluten/pull/6872#discussion_r1719256302
Do you think we can remove `GlutenByteInputStream` now and only maintain `VeloxInputStream`? I would suggest rename `VeloxInputStream` to `RssSortShuffleReaderInputStream` as it's only targeted for rss sort shuffle reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "kecookier (via GitHub)" <gi...@apache.org>.
kecookier commented on PR #12984:
URL: https://github.com/apache/gluten/pull/12984#issuecomment-5728916687
@marin-ma Agreed on both points. Will update the PR shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "kecookier (via GitHub)" <gi...@apache.org>.
kecookier commented on code in PR #12984:
URL: https://github.com/apache/gluten/pull/12984#discussion_r4044351667
##########
cpp/velox/shuffle/VeloxShuffleReader.cc:
##########
@@ -865,10 +911,18 @@ void VeloxRssSortShuffleReaderDeserializer::VeloxInputStream::next(bool throwIfP
GLUTEN_ASSIGN_OR_THROW(int64_t realBytes, in_->Read(readBytes, buffer_->asMutable<char>()));
if (realBytes > 0) {
offset_ = realBytes;
+ totalBytesRead_ += realBytes;
+ atEnd_ = false;
setRange({buffer_->asMutable<uint8_t>(), static_cast<int32_t>(realBytes), 0});
- } else if (throwIfPastEnd) {
- VELOX_FAIL(
- "Reading past end of VeloxRssSortShuffleReaderDeserializer::VeloxInputStream, real bytes = {}", realBytes);
+ } else {
Review Comment:
in_->Read() never returns a negative value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12985][VL] Fix multi-window page deserialization in rss-sort shuffle reader [gluten]
Posted by "kecookier (via GitHub)" <gi...@apache.org>.
kecookier commented on PR #12984:
URL: https://github.com/apache/gluten/pull/12984#issuecomment-5742019039
@marin-ma Both done. Could you take a look when you have a moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org