You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/30 01:16:57 UTC
[GitHub] [arrow] westonpace opened a new pull request, #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
westonpace opened a new pull request, #13036:
URL: https://github.com/apache/arrow/pull/13036
This builds on top of #13035 which is also important for avoiding segmentation faults. On top of that there were a few more problems:
* The python was using `SourceNodeOptions::FromTable` which is a rather dangerous method (mainly useful for unit testing) as it doesn't share ownership of the input table (even worse, it takes a const ref). Python was not keeping the table alive and it was maybe possible for the table to deleted out from under the plan (I'm not entirely sure this was causing issues but it seemed risky). I switched to TableSourceNode which shares ownership of the table (and is a bit more efficient).
* Setting use_threads to False did nothing because `_perform_join` was not passing the arg on to `execplan`.
* When fixing the above and running with `use_threads=False` it was creating a single thread executor but the current best practice is to pass in nullptr.
* Finally, the actual bug was my improper fix in #12894 . I had still left a small window open for `End` to be called between `Submit` and `AddTask` which would allow the task to be submitted but not participate in setting `finished` on the node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
westonpace commented on PR #13036:
URL: https://github.com/apache/arrow/pull/13036#issuecomment-1113884973
I was able to get it to reproduce in debug mode with `stress -c 32`. Then I got reasonable stack traces that showed the hash join node still working while the python was getting ready to start the next test. That was pretty similar to the debugging I did in #12894 which fortunately was still recent enough I remember roughly what was happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
URL: https://github.com/apache/arrow/pull/13036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13036:
URL: https://github.com/apache/arrow/pull/13036#issuecomment-1113883931
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13036:
URL: https://github.com/apache/arrow/pull/13036#issuecomment-1113885234
Aha.
Hmm, I wonder if it'd be useful to somehow have Pytest also assert that Arrow's thread pools are idle in between tests. (And frankly, Googletest as well.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13036:
URL: https://github.com/apache/arrow/pull/13036#issuecomment-1113883925
https://issues.apache.org/jira/browse/ARROW-16417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
westonpace commented on PR #13036:
URL: https://github.com/apache/arrow/pull/13036#issuecomment-1113885656
Technically, we aren't quite there yet. The async generators in the scanner are allowed to run some cleanup after the exec plan runs. However, they strongly capture all of their state. Once #12468 merges then I think this might be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13036: ARROW-16417: [C++][Python] Segfault in test_exec_plan.py / test_joins
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13036:
URL: https://github.com/apache/arrow/pull/13036#issuecomment-1119504498
Benchmark runs are scheduled for baseline = 760ad20067418e0c9a4d6bf524c52f50783bc135 and contender = 7809c6d7cb9dcf327840b0f9db1bff436d381f29. 7809c6d7cb9dcf327840b0f9db1bff436d381f29 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/08892a47d50741408503751417f8aad6...e82147137e1a4b4e86b23dd4dfcd4aa5/)
[Finished :arrow_down:1.98% :arrow_up:0.31%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/60022d0bd781464599e7bd41acca41c5...5b9f6858e7db4669be43832c69f005f4/)
[Finished :arrow_down:0.71% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/5e00d43064db427dbd1dfa6fceca03da...cff06a59011e4d03809bc1253fb7a5a9/)
[Finished :arrow_down:1.18% :arrow_up:0.08%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/41acf638bde7420eb03855e77e7a8f35...ccc22b5568cd4ff189a158c908947801/)
Buildkite builds:
[Finished] [`7809c6d7` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/680)
[Finished] [`7809c6d7` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/677)
[Finished] [`7809c6d7` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/666)
[Finished] [`7809c6d7` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/682)
[Finished] [`760ad200` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/679)
[Finished] [`760ad200` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/676)
[Finished] [`760ad200` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/665)
[Finished] [`760ad200` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/681)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org