You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/08 18:21:00 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #12845: [C++] Increase thread limit to work around thread issues
wjones127 opened a new pull request, #12845:
URL: https://github.com/apache/arrow/pull/12845
In #12339 we added one, which enabled joining one table to one dataset using `use_threads=false`. However, I found that joining two datasets hit the thread limit.
There are plans to find [a long-term fix](https://issues.apache.org/jira/browse/ARROW-16072) that can run these operations synchronously with fewer threads, but that won't be ready for the next release.
As a temporary fix for 8.0.0, I propose just bumping up the `local_states_` capacity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
westonpace commented on code in PR #12845:
URL: https://github.com/apache/arrow/pull/12845#discussion_r846429806
##########
cpp/src/arrow/compute/exec/hash_join.cc:
##########
@@ -103,7 +103,8 @@ class HashJoinBasicImpl : public HashJoinImpl {
filter_ = std::move(filter);
output_batch_callback_ = std::move(output_batch_callback);
finished_callback_ = std::move(finished_callback);
- local_states_.resize(num_threads + 1); // +1 for calling thread + worker threads
+ // Adding ten, since each side of join might have an IO thread being called from.
Review Comment:
Can you add `TODO(ARROW-15732)` to this comment so we know when to clean it up?
##########
cpp/src/arrow/compute/exec/hash_join.cc:
##########
@@ -103,7 +103,8 @@ class HashJoinBasicImpl : public HashJoinImpl {
filter_ = std::move(filter);
output_batch_callback_ = std::move(output_batch_callback);
finished_callback_ = std::move(finished_callback);
- local_states_.resize(num_threads + 1); // +1 for calling thread + worker threads
+ // Adding ten, since each side of join might have an IO thread being called from.
+ local_states_.resize(num_threads + 10);
Review Comment:
```suggestion
local_states_.resize(GetCpuThreadPoolCapacity() + io::GetIOThreadPoolCapacity() + 1);
```
This is slightly less arbitrary than 10 and we'd have to be doing something truly weird to go over this value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12845: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12845:
URL: https://github.com/apache/arrow/pull/12845#issuecomment-1093162670
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12845:
URL: https://github.com/apache/arrow/pull/12845#issuecomment-1093163573
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12845:
URL: https://github.com/apache/arrow/pull/12845#issuecomment-1093163527
https://issues.apache.org/jira/browse/ARROW-15718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
wjones127 commented on PR #12845:
URL: https://github.com/apache/arrow/pull/12845#issuecomment-1101761551
It looks like that Java failure is simply flaky right now. See other recent failures:
* https://github.com/apache/arrow/runs/6061027109?check_suite_focus=true
* https://github.com/apache/arrow/runs/6062605054?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #12845:
URL: https://github.com/apache/arrow/pull/12845#issuecomment-1107452661
Benchmark runs are scheduled for baseline = 1763622b6f60e974e495b3349cc2f4b7caaf1951 and contender = 08ab8b0fdb63a0e864d6175ff417bca038034e62. 08ab8b0fdb63a0e864d6175ff417bca038034e62 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/dd19e80e53df418eacc6c1e9e94360ab...e14d41b1a27a4d9d88ad42225d41a1c7/)
[Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/152f7dc5024a44e8801f1fefef725a7e...6b543a90a3e240bcb3a2d7b65ea633c4/)
[Failed :arrow_down:0.75% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/2816745be03f4b59b109515ff85b8735...44dfa2c5b81e4f24805cd644c9430c81/)
[Finished :arrow_down:0.67% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/73f4539b4a5f4dd0a83cc70b26dce91c...7a49dc50a6384297b3b48e915da38b78/)
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/566| `08ab8b0f` ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/554| `08ab8b0f` test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/552| `08ab8b0f` ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/564| `08ab8b0f` ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/565| `1763622b` ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/553| `1763622b` test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/551| `1763622b` ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/563| `1763622b` ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace closed pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
Posted by GitBox <gi...@apache.org>.
westonpace closed pull request #12845: ARROW-15718: [C++] Increase thread limit to work around thread issues
URL: https://github.com/apache/arrow/pull/12845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org