You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2025/07/24 12:25:45 UTC
[PR] GH-47184: [Parquet][C++] Try prevent from too large buffer when using FixedSizeBuilder [arrow]
mapleFU opened a new pull request, #47185:
URL: https://github.com/apache/arrow/pull/47185
### Rationale for this change
See: https://github.com/apache/arrow/issues/47184
### What changes are included in this PR?
This patch add more checkings directly in builder. Maybe we can add more checking in this scenerio.
### Are these changes tested?
By https://github.com/apache/arrow-testing/pull/110
### Are there any user-facing changes?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2314810757
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
@pitrou gentle ping for idea here, should I prevent overflow from `ReserveValues` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3270999510
gentle ping @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3220813526
> I've change the checking from Builder ( which is commonly used ) to only DeltaByteArrayDecoder ( which is only place can cause FLBA to have this problem)
Did you forget to push your changes?
(also, please rebase :-))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2299511855
##########
cpp/src/parquet/decoder.cc:
##########
@@ -2072,6 +2072,15 @@ class DeltaByteArrayDecoderImpl : public TypedDecoderImpl<DType> {
std::vector<ByteArray> values(num_values);
const int num_valid_values = GetInternal(values.data(), num_values - null_count);
DCHECK_EQ(num_values - null_count, num_valid_values);
+ if constexpr (std::is_same_v<DType, FLBAType>) {
+ // Checks all values
Review Comment:
Would try to put in the end of `GetInternal`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Try prevent from too large buffer when using FixedSizeBuilder [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2229861486
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,11 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
+ if (ARROW_PREDICT_FALSE(MultiplyWithOverflow(capacity, byte_width_, &dest_capacity_bytes))) {
+ return Status::Invalid("Resize: capacity too large for byte width");
Review Comment:
It may be better that we also show the requested capacity like https://github.com/apache/arrow/blob/25f8f008061b978137a2c1e5bc934d07fae56e3e/cpp/src/arrow/array/builder_base.h#L286-L294 does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3220623722
I've change the checking from Builder ( which is commonly used ) to only DeltaByteArrayDecoder ( which is only place can cause FLBA to have this problem)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2298588907
##########
cpp/src/parquet/decoder.cc:
##########
@@ -2072,6 +2072,15 @@ class DeltaByteArrayDecoderImpl : public TypedDecoderImpl<DType> {
std::vector<ByteArray> values(num_values);
const int num_valid_values = GetInternal(values.data(), num_values - null_count);
DCHECK_EQ(num_values - null_count, num_valid_values);
+ if constexpr (std::is_same_v<DType, FLBAType>) {
+ // Checks all values
Review Comment:
Should we do this in `GetInternal` instead? It will then benefit the `Decode` method as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2298592822
##########
cpp/src/parquet/decoder.cc:
##########
@@ -2072,6 +2072,15 @@ class DeltaByteArrayDecoderImpl : public TypedDecoderImpl<DType> {
std::vector<ByteArray> values(num_values);
const int num_valid_values = GetInternal(values.data(), num_values - null_count);
DCHECK_EQ(num_values - null_count, num_valid_values);
+ if constexpr (std::is_same_v<DType, FLBAType>) {
Review Comment:
Good point, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3299652916
Thanks again for fixing this @mapleFU !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2294967941
##########
cpp/src/parquet/decoder.cc:
##########
@@ -207,6 +207,10 @@ struct ArrowBinaryHelper<FLBAType, ::arrow::FixedSizeBinaryType> {
Status AppendValue(const uint8_t* data, int32_t length,
std::optional<int64_t> estimated_remaining_data_length = {}) {
+ if (ARROW_PREDICT_FALSE(length != acc_->byte_width())) {
Review Comment:
I don't know can we decrease overhead for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3215934038
Don't know why cannot reproduce test failed on MacOS M1...🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3299507734
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Try prevent from too large buffer when using FixedSizeBuilder [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2285670230
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
+ if (ARROW_PREDICT_FALSE(
+ MultiplyWithOverflow(capacity, byte_width_, &dest_capacity_bytes))) {
+ return Status::Invalid("Resize: capacity too large for byte width (requested: ",
Review Comment:
We have `Status::CapacityError` for such errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2298585976
##########
cpp/src/parquet/decoder.cc:
##########
@@ -2072,6 +2072,15 @@ class DeltaByteArrayDecoderImpl : public TypedDecoderImpl<DType> {
std::vector<ByteArray> values(num_values);
const int num_valid_values = GetInternal(values.data(), num_values - null_count);
DCHECK_EQ(num_values - null_count, num_valid_values);
+ if constexpr (std::is_same_v<DType, FLBAType>) {
Review Comment:
Delta uses DeltaLength for Encoding, but DeltaLength it self doesn't supports FLBA, since it's for var-length types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3299537576
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2299511079
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
I don't know, I'll remove this and have a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2300868963
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
Oh, it's because `ReserveValues` is called above this:
```c++
void ReserveValues(int64_t extra_values) override {
ARROW_DCHECK(!uses_values_);
TypedRecordReader::ReserveValues(extra_values);
PARQUET_THROW_NOT_OK(array_builder_.Reserve(extra_values));
}
```
`!uses_values_` is only called by ByteArray and FLBA, so other types doesn't touches this case... And ByteArray would not have a multiplier, which is less possible to touch this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2300868963
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
Oh, it's because `ReserveValues` is called above this:
```c++
void ReserveValues(int64_t extra_values) override {
ARROW_DCHECK(!uses_values_);
TypedRecordReader::ReserveValues(extra_values);
PARQUET_THROW_NOT_OK(array_builder_.Reserve(extra_values));
}
```
`!uses_values_` is only called by ByteArray and FLBA, so other types doesn't touches this case...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2298583259
##########
cpp/src/parquet/decoder.cc:
##########
@@ -2072,6 +2072,15 @@ class DeltaByteArrayDecoderImpl : public TypedDecoderImpl<DType> {
std::vector<ByteArray> values(num_values);
const int num_valid_values = GetInternal(values.data(), num_values - null_count);
DCHECK_EQ(num_values - null_count, num_valid_values);
+ if constexpr (std::is_same_v<DType, FLBAType>) {
Review Comment:
Should `DeltaLengthByteArrayDecoder` also get the same checks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2298581662
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
Do we still need this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3299549586
Revision: 08b020b24b2c00e862473c2f00c0bb50b7a4e374
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f7e0ba7b53](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f7e0ba7b53)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711258/job/50512712974)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/17772710842/job/50512711434)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711479/job/50512713410)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711091/job/50512712175)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711257/job/50512712654)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711505/job/50512713650)|
|test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711379/job/50512713040)|
|test-debian-12-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/17772710845/job/50512711143)|
|test-debian-12-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/17772710829/job/50512711212)|
|test-fedora-42-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711063/job/50512712187)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772710779/job/50512711123)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711515/job/50512713517)|
|test-ubuntu-22.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711294/job/50512712713)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711207/job/50512712890)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711499/job/50512713518)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772710905/job/50512711717)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711107/job/50512712129)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711135/job/50512712189)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711746/job/50512714320)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711217/job/50512712449)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/17772711071/job/50512712583)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2295441589
##########
cpp/src/parquet/decoder.cc:
##########
@@ -207,6 +207,10 @@ struct ArrowBinaryHelper<FLBAType, ::arrow::FixedSizeBinaryType> {
Status AppendValue(const uint8_t* data, int32_t length,
std::optional<int64_t> estimated_remaining_data_length = {}) {
+ if (ARROW_PREDICT_FALSE(length != acc_->byte_width())) {
+ return Status::Invalid("FLBAType requires fixed-length values length to be ",
+ acc_->byte_width(), "but got ", length);
Review Comment:
```suggestion
return Status::Invalid("FLBA type requires fixed-length ",
acc_->byte_width(), " but got ", length);
```
##########
cpp/src/parquet/decoder.cc:
##########
@@ -207,6 +207,10 @@ struct ArrowBinaryHelper<FLBAType, ::arrow::FixedSizeBinaryType> {
Status AppendValue(const uint8_t* data, int32_t length,
std::optional<int64_t> estimated_remaining_data_length = {}) {
+ if (ARROW_PREDICT_FALSE(length != acc_->byte_width())) {
Review Comment:
Did the inequality happen in the fuzz test?
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
+ if (ARROW_PREDICT_FALSE(
+ MultiplyWithOverflow(capacity, byte_width_, &dest_capacity_bytes))) {
+ return Status::CapacityError("Resize: capacity too large for byte width (requested: ",
Review Comment:
```suggestion
return Status::CapacityError("Resize: capacity overflows (requested: ",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Try prevent from too large buffer when using FixedSizeBuilder [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3113280450
:warning: GitHub issue #47184 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2300791249
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
I suggest we need this change, I add cout and run fuzzing and found there're so many error message now:
```
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
int64_t dest_capacity_bytes;
if (ARROW_PREDICT_FALSE(
MultiplyWithOverflow(capacity, byte_width_, &dest_capacity_bytes))) {
std::cout << "Resize: capacity overflows (requested: " << capacity <<
", byte_width: " << byte_width_ << std::endl;
return Status::CapacityError("Resize: capacity overflows (requested: ", capacity,
", byte_width: ", byte_width_, ")");
}
RETURN_NOT_OK(byte_builder_.Resize(dest_capacity_bytes));
return ArrayBuilder::Resize(capacity);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2300843804
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
Perhaps the problem here is:
```c++
Status ReadColumn(int i, const std::vector<int>& row_groups, ColumnReader* reader,
std::shared_ptr<ChunkedArray>* out) {
BEGIN_PARQUET_CATCH_EXCEPTIONS
// TODO(wesm): This calculation doesn't make much sense when we have repeated
// schema nodes
int64_t records_to_read = 0;
for (auto row_group : row_groups) {
// Can throw exception
std::cout << "Numvalues:" << reader_->metadata()->RowGroup(row_group)->ColumnChunk(i)->num_values() << '\n';
records_to_read +=
reader_->metadata()->RowGroup(row_group)->ColumnChunk(i)->num_values();
}
```
1. Num values is `int64_t::max()`
2. Generally, builder will throw exception. For fixed sized type, the value would exceeds the bounds
Perhaps for builder, the FLBA column is just so huge, and causes the issue. The integer and float types might also has this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2348466828
##########
cpp/src/arrow/array/builder_binary.cc:
##########
@@ -162,7 +164,13 @@ void FixedSizeBinaryBuilder::Reset() {
Status FixedSizeBinaryBuilder::Resize(int64_t capacity) {
RETURN_NOT_OK(CheckCapacity(capacity));
- RETURN_NOT_OK(byte_builder_.Resize(capacity * byte_width_));
+ int64_t dest_capacity_bytes;
Review Comment:
Yes, I think this change is ok. Sorry for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3299516068
Revision: 078b5743ed1c5f8b7a4448f877584003ea945bd9
Submitted crossbow builds: [ursacomputing/crossbow @ actions-0e5c2a50fe](https://github.com/ursacomputing/crossbow/branches/all?query=actions-0e5c2a50fe)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480000/job/50511904264)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479296/job/50511903041)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480550/job/50511906038)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479988/job/50511904248)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480356/job/50511905508)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479351/job/50511903121)|
|test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479680/job/50511903928)|
|test-debian-12-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479645/job/50511903935)|
|test-debian-12-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479967/job/50511904206)|
|test-fedora-42-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480045/job/50511904435)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480221/job/50511905033)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480534/job/50511906085)|
|test-ubuntu-22.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480202/job/50511905094)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480039/job/50511904441)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479290/job/50511903039)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479557/job/50511903604)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480091/job/50511904584)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479595/job/50511903620)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/17772480255/job/50511905176)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479455/job/50511903416)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/17772479283/job/50511903436)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3300693601
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 37dcfccd65c1d9e51ab000edf91768526f13965a.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/50542945412) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #47185:
URL: https://github.com/apache/arrow/pull/47185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #47185:
URL: https://github.com/apache/arrow/pull/47185#discussion_r2296100339
##########
cpp/src/parquet/decoder.cc:
##########
@@ -207,6 +207,10 @@ struct ArrowBinaryHelper<FLBAType, ::arrow::FixedSizeBinaryType> {
Status AppendValue(const uint8_t* data, int32_t length,
std::optional<int64_t> estimated_remaining_data_length = {}) {
+ if (ARROW_PREDICT_FALSE(length != acc_->byte_width())) {
Review Comment:
See https://github.com/apache/arrow/actions/runs/17131673477/job/48597348959
The fuzzer will generate random underlying data from seeds, and this caused a heap buffer overflow when type mismatches. Maybe a boundary check on buffer is more lightweight, which only prevent from heap buffer overflow.
Size check would enforce underlying data to be same sized. However it introduce a size checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47184: [Parquet][C++] Avoid multiplication overflow in FixedSizeBinaryBuilder::Reserve [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47185:
URL: https://github.com/apache/arrow/pull/47185#issuecomment-3219567888
Can you run the Parquet benchmarks to see if there any regressions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org