You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "benibus (via GitHub)" <gi...@apache.org> on 2023/04/11 17:36:05 UTC
[GitHub] [arrow] benibus opened a new pull request, #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
benibus opened a new pull request, #35047:
URL: https://github.com/apache/arrow/pull/35047
### What changes are included in this PR?
Increases the block size used in the `ScanWithParallelDecoding` test to reduce the number of (potentially parallel) parsing/decoding jobs from 1000+ to roughly 60 while increasing the runtime of each job. This should still satisfy the purpose of test without going completely over the top.
### Are these changes tested?
Yes, tested locally on the alpine docker image many times after successfully reproducing the original issue.
### Are there any user-facing changes?
No
### Notes
This doesn't solve the underlying cause (although the testing parameters were arguably far too unusual in the first place), however I do believe that I've identified the issue via a core dump.
The problem starts [here](https://github.com/apache/arrow/blob/47a602dbd9b7b7f7720a5e62467e3e6c61712cf3/cpp/src/arrow/json/reader.cc#L362-L369), where a `MappingGenerator` gets stacked on top of a generator that applies readahead. It seems that the underlying futures were completing very quickly, resulting in `AddCallback` being called recursively many, many times - starting [here](https://github.com/apache/arrow/blob/47a602dbd9b7b7f7720a5e62467e3e6c61712cf3/cpp/src/arrow/util/async_generator.h#L240). This leads to a stack overflow under specific circumstances.
So, to fully guard against the problem, you'd probably want to change the logic of `MappingGenerator` to use `TryAddCallback` + an inner loop to avoid overflowing the stack. Not entirely sure if doing this would be worthwhile though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1503965120
@github-actions crossbow submit test-alpine-linux-cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] benibus commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "benibus (via GitHub)" <gi...@apache.org>.
benibus commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1505546898
> Which mapping generator? Is this one of the ones used in the scanner?
No, the one in the JSON reader. The reader uses two `MappingGenerator`s - one to parse/decode incoming buffers, then another to "unwrap" the final result and return the record batch to the caller. The latter was the one causing problems (I think because readahead was being applied in between, so finished futures were being buffered).
That being said, the overflow only seemed to occur with the new scanner, but I suspect it's not really "at fault" of anything - just a difference in implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1503824265
* Closes: #34653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1507826309
Benchmark runs are scheduled for baseline = 61203456ed33268df0c8c164348a203c7c1be8ca and contender = 5e8db3156c733a31e196683011db113e76ce6a32. 5e8db3156c733a31e196683011db113e76ce6a32 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/b357ae6edce2492d9243d8fc412c8a1a...9d5ac7ece5774f0bb357292ea00cc20d/)
[Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c71b8f0902c3405d827971e4b0843028...82e6d01059b444f89fae537c51952144/)
[Finished :arrow_down:1.53% :arrow_up:1.02%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/533dd7d23af149b096b13b771d1cfa7e...f405fa51a8904d89a0b77e2f62878e5a/)
[Finished :arrow_down:0.28% :arrow_up:0.06%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/8f7589a9484b41a5b30d6025abfca367...f895f110275e44d5a9f9fc5d1cf7c4e9/)
Buildkite builds:
[Finished] [`5e8db315` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2691)
[Failed] [`5e8db315` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2725)
[Finished] [`5e8db315` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2689)
[Finished] [`5e8db315` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2716)
[Finished] [`61203456` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2690)
[Failed] [`61203456` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2724)
[Finished] [`61203456` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2688)
[Finished] [`61203456` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2715)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1507826891
['Python', 'R'] benchmarks have high level of regressions.
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/533dd7d23af149b096b13b771d1cfa7e...f405fa51a8904d89a0b77e2f62878e5a/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1503967702
Revision: 01e1cf29799ca9eeb517b2afbf4ac7f3b80e2cd5
Submitted crossbow builds: [ursacomputing/crossbow @ actions-d0a3bb929b](https://github.com/ursacomputing/crossbow/branches/all?query=actions-d0a3bb929b)
|Task|Status|
|----|------|
|test-alpine-linux-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/4671181740/jobs/8271827639)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace merged pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace merged PR #35047:
URL: https://github.com/apache/arrow/pull/35047
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #35047: GH-34653: [CI][C++] Fix for arrow-dataset-file-json-test segfault on alpine-linux-cpp
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #35047:
URL: https://github.com/apache/arrow/pull/35047#issuecomment-1505296790
:tada: thanks for your persistence in fixing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org