You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2026/04/09 15:28:22 UTC
[PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
pitrou opened a new pull request, #49698:
URL: https://github.com/apache/arrow/pull/49698
…
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
* [AI-generated Code Guidance](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code)
Please remove this line and the above text before creating your pull request.
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
**This PR contains a "Critical Fix".** (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4215802168
Revision: 44711e307c799174af109f37d3240ab40d918869
Submitted crossbow builds: [ursacomputing/crossbow @ actions-cc0beb58ed](https://github.com/ursacomputing/crossbow/branches/all?query=actions-cc0beb58ed)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262214/job/70645186272)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262446/job/70645187337)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263139/job/70645189495)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263053/job/70645189016)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263640/job/70645191495)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262696/job/70645187760)|
|test-debian-13-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262371/job/70645186474)|
|test-debian-13-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/24201264054/job/70645194130)|
|test-debian-experimental-cpp-gcc-15|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263171/job/70645189529)|
|test-fedora-42-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263962/job/70645192042)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262181/job/70645186273)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262842/job/70645188452)|
|test-ubuntu-22.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263398/job/70645191771)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262875/job/70645189086)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263652/job/70645191841)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263415/job/70645192427)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262465/job/70645186794)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/24201264842/job/70645196193)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/24201262750/job/70645189041)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263713/job/70645192789)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/24201263662/job/70645191141)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4236461778
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #49698:
URL: https://github.com/apache/arrow/pull/49698#discussion_r3073353971
##########
cpp/src/arrow/ipc/reader.cc:
##########
@@ -613,16 +614,90 @@ Status DecompressBuffers(Compression::type compression, const IpcReadOptions& op
});
}
+// Helper class to run post-ArrayLoader steps:
+// buffer decompression, dictionary resolution, buffer re-alignment.
+struct RecordBatchLoader {
+ Result<std::shared_ptr<RecordBatch>> CreateRecordBatch(ArrayDataVector columns) {
+ ARROW_ASSIGN_OR_RAISE(auto filtered_columns, CreateColumns(std::move(columns)));
+
+ std::shared_ptr<Schema> filtered_schema;
+ if (!inclusion_mask_.empty()) {
+ FieldVector filtered_fields;
+ for (int i = 0; i < schema_->num_fields(); ++i) {
+ if (inclusion_mask_[i]) {
+ filtered_fields.push_back(schema_->field(i));
+ }
+ }
+ filtered_schema = schema(std::move(filtered_fields), schema_->metadata());
+ } else {
+ filtered_schema = schema_;
+ }
+
+ return RecordBatch::Make(std::move(filtered_schema), batch_length_,
+ std::move(filtered_columns));
+ }
+
+ Result<ArrayDataVector> CreateColumns(ArrayDataVector columns,
+ bool resolve_dictionaries = true) {
+ if (resolve_dictionaries) {
+ // Dictionary resolution needs to happen on the unfiltered columns,
+ // because fields are mapped structurally (by path in the original schema).
+ RETURN_NOT_OK(ResolveDictionaries(columns, *context_.dictionary_memo,
+ context_.options.memory_pool));
+ }
+
+ ArrayDataVector filtered_columns;
+ if (!inclusion_mask_.empty()) {
+ FieldVector filtered_fields;
+ for (int i = 0; i < schema_->num_fields(); ++i) {
+ if (inclusion_mask_[i]) {
+ filtered_columns.push_back(std::move(columns[i]));
Review Comment:
nit: given the code is more encapsulated from its users now, is it worth DCHECKing that the column is not-nullptr and actually got loaded here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4215784527
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4236762203
@raulcd This may be a good last-minute fix for 24.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4236719334
@lidavidm Could you perhaps take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #49698:
URL: https://github.com/apache/arrow/pull/49698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4239860135
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 982797cd20c58173ece1132e18a1a59d3e89b7ca.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/71167096520) has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outcome [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49698:
URL: https://github.com/apache/arrow/pull/49698#issuecomment-4236477925
Revision: 3b2d1014810f38ec4f0080e3eab0dfd75440e110
Submitted crossbow builds: [ursacomputing/crossbow @ actions-265034e4bd](https://github.com/ursacomputing/crossbow/branches/all?query=actions-265034e4bd)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108647/job/71080230859)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109996/job/71080235382)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109769/job/71080234777)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108823/job/71080231384)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108850/job/71080231585)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108301/job/71080229948)|
|test-debian-13-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/24344110199/job/71080235972)|
|test-debian-13-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109232/job/71080233013)|
|test-debian-experimental-cpp-gcc-15|[](https://github.com/ursacomputing/crossbow/actions/runs/24344110217/job/71080236113)|
|test-fedora-42-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109415/job/71080233419)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109335/job/71080233308)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109174/job/71080232500)|
|test-ubuntu-22.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109350/job/71080233181)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108302/job/71080229941)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109608/job/71080234093)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/24344110059/job/71080235461)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108321/job/71080230093)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108993/job/71080231881)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108416/job/71080230190)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/24344108730/job/71080231696)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/24344109801/job/71080234725)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org