You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/05 23:10:58 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #15213: GH-15212: [C++] ORC writer doesn't work on sliced list arrays
wjones127 opened a new pull request, #15213:
URL: https://github.com/apache/arrow/pull/15213
Writing sliced list and map arrays will produce incorrect data. This happens because the arrays are recreated using `BaseListArray.values()`, which doesn't account for the offsets, and the offsets was not passed. To fix, I changed to pass along the offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #15213: GH-15212: [C++] ORC writer doesn't work on sliced list arrays
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15213:
URL: https://github.com/apache/arrow/pull/15213#issuecomment-1372911323
* Closes: #15212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #15213: GH-15212: [C++] ORC writer doesn't work on sliced list arrays
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #15213:
URL: https://github.com/apache/arrow/pull/15213#issuecomment-1385735815
Can we rephrase the PR title to describe what the PR does? "XXX doesn't work" is ambiguous (does it describe what the PR is supposed to fix?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 merged pull request #15213: GH-15212: [C++] fix sliced list array writing in ORC
Posted by GitBox <gi...@apache.org>.
wjones127 merged PR #15213:
URL: https://github.com/apache/arrow/pull/15213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #15213: GH-15212: [C++] ORC writer doesn't work on sliced list arrays
Posted by GitBox <gi...@apache.org>.
wjones127 commented on PR #15213:
URL: https://github.com/apache/arrow/pull/15213#issuecomment-1374020706
Don't merge this. I think there may also be issues with StringArray as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #15213: GH-15212: [C++] ORC writer doesn't work on sliced list arrays
Posted by GitBox <gi...@apache.org>.
wgtmac commented on code in PR #15213:
URL: https://github.com/apache/arrow/pull/15213#discussion_r1063118195
##########
cpp/src/arrow/adapters/orc/adapter_test.cc:
##########
@@ -536,6 +536,15 @@ TEST_F(TestORCWriterTrivialNoConversion, writeFilledChunkAndSelectField) {
&selected_indices);
}
+TEST_F(TestORCWriterTrivialNoConversion, writeSlicedBatch) {
+ std::shared_ptr<Table> table =
+ GenerateRandomTable(table_schema, /*size=*/100, /*min_num_chunks=*/1,
Review Comment:
Does it cover type like `list<list<X>>` to make sure that offset of the inner list is correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #15213: GH-15212: [C++] ORC writer doesn't work on sliced list arrays
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #15213:
URL: https://github.com/apache/arrow/pull/15213#discussion_r1063568116
##########
cpp/src/arrow/adapters/orc/adapter_test.cc:
##########
@@ -536,6 +536,15 @@ TEST_F(TestORCWriterTrivialNoConversion, writeFilledChunkAndSelectField) {
&selected_indices);
}
+TEST_F(TestORCWriterTrivialNoConversion, writeSlicedBatch) {
+ std::shared_ptr<Table> table =
+ GenerateRandomTable(table_schema, /*size=*/100, /*min_num_chunks=*/1,
Review Comment:
The test doesn't cover that case. But it should be fine, since `NormalizeArray` is recursive. So `NormalizeArray(list<list<X>>)` calls `NormalizeArray(list<X>)`, which calls `NormalizeArray(X)`. The offsets at each level should be handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #15213: GH-15212: [C++] fix sliced list array writing in ORC
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #15213:
URL: https://github.com/apache/arrow/pull/15213#issuecomment-1396971385
Benchmark runs are scheduled for baseline = 2b50694c10e09e4a1343b62c6b5f44ad4403d0e1 and contender = 04ffb1f740f1e868c08313fa9043070345d9b6f0. 04ffb1f740f1e868c08313fa9043070345d9b6f0 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/b69a7118eddb41a695cd537131c0bcf4...f6fd4fd5f7a94999a82da3fa4141e390/)
[Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/e62713d5ba6e4cbda2b0f634cc137de1...b902fb2824fa49c3a6ea24b4ab4219f4/)
[Finished :arrow_down:0.51% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8f9ba14983c34891ad8653cf430cb4b3...259585af4f5049a7a31029ea64f09ed0/)
[Finished :arrow_down:0.22% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/895eb1a43a284e71b5d818b1ad9d2986...19a391a76cbc4f778ed1f9c1afe42db9/)
Buildkite builds:
[Finished] [`04ffb1f7` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2215)
[Failed] [`04ffb1f7` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2241)
[Finished] [`04ffb1f7` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2211)
[Finished] [`04ffb1f7` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2234)
[Finished] [`2b50694c` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2214)
[Failed] [`2b50694c` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2240)
[Finished] [`2b50694c` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2210)
[Finished] [`2b50694c` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2233)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org