You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/07 04:39:49 UTC
[GitHub] [arrow] michalursa opened a new pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
michalursa opened a new pull request #11350:
URL: https://github.com/apache/arrow/pull/11350
Fixing 3 issues:
- one in SchemaProjectionMaps - I simplified all of the code to get rid of thread synchronization at all
- one in TaskScheduler - added (unnecessary) mutex
- one in HashJoinImpl - switching from shared byte vector to local bit vectors and merge (for recording if a match for a hash table row has been seen)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946350905
> @michalursa Can we merge this?
@kou Yes, I believe so. This is a relatively simple change (more precisely 3 simple fixes put together) and I don't see any errors that could be related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on a change in pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r731241387
##########
File path: cpp/src/arrow/compute/exec/hash_join.cc
##########
@@ -636,6 +637,47 @@ class HashJoinBasicImpl : public HashJoinImpl {
return ScanHashTable(thread_index);
}
+ void InitHasMatchIfNeeded(ThreadLocalState* local_state) {
+ if (local_state->is_has_match_initialized) {
+ return;
+ }
+ if (!hash_table_empty_) {
+ int32_t num_rows = hash_table_keys_.num_rows();
+ local_state->has_match.resize(BitUtil::BytesForBits(num_rows));
+ memset(local_state->has_match.data(), 0, BitUtil::BytesForBits(num_rows));
+ }
+ local_state->is_has_match_initialized = true;
+ }
+
+ void MergeHasMatch() {
+ if (hash_table_empty_) {
+ return;
+ }
+
+ int32_t num_rows = hash_table_keys_.num_rows();
+ has_match_.resize(BitUtil::BytesForBits(num_rows));
+ memset(has_match_.data(), 0, BitUtil::BytesForBits(num_rows));
+
+ for (size_t tid = 0; tid < local_states_.size(); ++tid) {
+ if (!local_states_[tid].is_initialized) {
+ continue;
+ }
+ if (!local_states_[tid].is_has_match_initialized) {
+ continue;
+ }
+ uint8_t* dst8 = has_match_.data();
+ const uint8_t* src8 = local_states_[tid].has_match.data();
+ uint64_t* dst64 = reinterpret_cast<uint64_t*>(dst8);
+ const uint64_t* src64 = reinterpret_cast<const uint64_t*>(src8);
+ for (int32_t i = 0; i < num_rows / 64; ++i) {
+ dst64[i] |= src64[i];
+ }
+ for (int32_t i = num_rows / 64 * 8; i < BitUtil::CeilDiv(num_rows, 8); ++i) {
+ dst8[i] |= src8[i];
+ }
Review comment:
Thanks, I made the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946165859
Revision: 9fd1c024392207bc3622da6a3307c06272efbc4c
Submitted crossbow builds: [ursacomputing/crossbow @ actions-994](https://github.com/ursacomputing/crossbow/branches/all?query=actions-994)
|Task|Status|
|----|------|
|conda-linux-gcc-py36-cpu-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-conda-linux-gcc-py36-cpu-r40)|
|conda-linux-gcc-py37-cpu-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-conda-linux-gcc-py37-cpu-r41)|
|conda-osx-clang-py36-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-conda-osx-clang-py36-r40)|
|conda-osx-clang-py37-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-conda-osx-clang-py37-r41)|
|conda-win-vs2017-py36-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-conda-win-vs2017-py36-r40)|
|conda-win-vs2017-py37-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-conda-win-vs2017-py37-r41)|
|homebrew-r-autobrew|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-homebrew-r-autobrew)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-fedora-33-cpp)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-arrow-backwards-compatibility)|
|test-r-depsource-auto|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-depsource-auto)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-depsource-system)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-devdocs)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-gcc-11)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-install-local)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-linux-as-cran)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-linux-rchk)|
|test-r-linux-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-linux-valgrind)|
|test-r-minimal-build|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-minimal-build)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-offline-maximal)|
|test-r-offline-minimal|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-offline-minimal)|
|test-r-rhub-debian-gcc-devel-lto-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rhub-debian-gcc-devel-lto-latest)|
|test-r-rhub-ubuntu-gcc-release-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rhub-ubuntu-gcc-release-latest)|
|test-r-rocker-r-base-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rocker-r-base-latest)|
|test-r-rstudio-r-base-3.6-bionic|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rstudio-r-base-3.6-bionic)|
|test-r-rstudio-r-base-3.6-centos7-devtoolset-8|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rstudio-r-base-3.6-centos7-devtoolset-8)|
|test-r-rstudio-r-base-3.6-centos8|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rstudio-r-base-3.6-centos8)|
|test-r-rstudio-r-base-3.6-opensuse15|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rstudio-r-base-3.6-opensuse15)|
|test-r-rstudio-r-base-3.6-opensuse42|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-r-rstudio-r-base-3.6-opensuse42)|
|test-r-ubuntu-21.04|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-ubuntu-21.04)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-r-versions)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-18.04-r-sanitizer|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-994-azure-test-ubuntu-18.04-r-sanitizer)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-994-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-940464232
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r723927168
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
Review comment:
For the record, we try to avoid including the `api.h` headers in internal Arrow code, as they tend to blow up compile times. Instead, one should only include the strictly necessary.
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/csv/api.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/api.h>
+#include <arrow/io/api.h>
+#include <arrow/io/compressed.h>
+#include <arrow/ipc/api.h>
+#include <arrow/pretty_print.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+#include <gmock/gmock-matchers.h>
+
+#include <iostream>
+
+namespace arrow {
+
+Result<std::shared_ptr<dataset::Dataset>> MakeDataset(const std::string& filename) {
+ auto fs = std::make_shared<fs::LocalFileSystem>();
+ auto fs_dataset_options = arrow::dataset::FileSystemFactoryOptions();
+ auto format = std::make_shared<arrow::dataset::IpcFileFormat>();
+ ARROW_ASSIGN_OR_RAISE(auto dataset_factory,
+ arrow::dataset::FileSystemDatasetFactory::Make(
+ fs, {filename}, format, fs_dataset_options));
+ auto finish_options = arrow::dataset::FinishOptions();
+ finish_options.validate_fragments = false;
+
+ auto inspect_options = arrow::dataset::InspectOptions();
+ inspect_options.fragments = 1;
+ finish_options.inspect_options = inspect_options;
+ return dataset_factory->Finish(finish_options);
+}
+
+Result<std::shared_ptr<dataset::ScanOptions>> DefaultOptions(
+ std::shared_ptr<dataset::Dataset> dataset) {
+ auto scanner_builder = std::make_shared<dataset::ScannerBuilder>(std::move(dataset));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseAsync(true));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseThreads(true));
+ ARROW_ASSIGN_OR_RAISE(auto scanner, scanner_builder->Finish());
+ return scanner->options();
+}
+
+Status RunMain() {
+ dataset::internal::Initialize();
+ ARROW_ASSIGN_OR_RAISE(
+ auto line_items_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/lineitem_1.uncompressed.feather"));
+ ARROW_ASSIGN_OR_RAISE(auto line_items_scan_opts, DefaultOptions(line_items_dataset));
+ ARROW_ASSIGN_OR_RAISE(
+ auto orders_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/orders_1.uncompressed.feather"));
Review comment:
Should these files go into https://github.com/apache/arrow-testing/ ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-940464765
Revision: f1dae79664dc0be394e0544b62ef79d7e1e34685
Submitted crossbow builds: [ursacomputing/crossbow @ actions-914](https://github.com/ursacomputing/crossbow/branches/all?query=actions-914)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-914-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-914-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r723927168
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
Review comment:
For the record, we try to avoid including the `api.h` headers in internal Arrow code, as they tend to blow up compile times. Instead, one should only include the strictly necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937439647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946267294
@michalursa Can we merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r723927899
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/csv/api.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/api.h>
+#include <arrow/io/api.h>
+#include <arrow/io/compressed.h>
+#include <arrow/ipc/api.h>
+#include <arrow/pretty_print.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+#include <gmock/gmock-matchers.h>
+
+#include <iostream>
+
+namespace arrow {
+
+Result<std::shared_ptr<dataset::Dataset>> MakeDataset(const std::string& filename) {
+ auto fs = std::make_shared<fs::LocalFileSystem>();
+ auto fs_dataset_options = arrow::dataset::FileSystemFactoryOptions();
+ auto format = std::make_shared<arrow::dataset::IpcFileFormat>();
+ ARROW_ASSIGN_OR_RAISE(auto dataset_factory,
+ arrow::dataset::FileSystemDatasetFactory::Make(
+ fs, {filename}, format, fs_dataset_options));
+ auto finish_options = arrow::dataset::FinishOptions();
+ finish_options.validate_fragments = false;
+
+ auto inspect_options = arrow::dataset::InspectOptions();
+ inspect_options.fragments = 1;
+ finish_options.inspect_options = inspect_options;
+ return dataset_factory->Finish(finish_options);
+}
+
+Result<std::shared_ptr<dataset::ScanOptions>> DefaultOptions(
+ std::shared_ptr<dataset::Dataset> dataset) {
+ auto scanner_builder = std::make_shared<dataset::ScannerBuilder>(std::move(dataset));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseAsync(true));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseThreads(true));
+ ARROW_ASSIGN_OR_RAISE(auto scanner, scanner_builder->Finish());
+ return scanner->options();
+}
+
+Status RunMain() {
+ dataset::internal::Initialize();
+ ARROW_ASSIGN_OR_RAISE(
+ auto line_items_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/lineitem_1.uncompressed.feather"));
+ ARROW_ASSIGN_OR_RAISE(auto line_items_scan_opts, DefaultOptions(line_items_dataset));
+ ARROW_ASSIGN_OR_RAISE(
+ auto orders_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/orders_1.uncompressed.feather"));
Review comment:
Should these files go into https://github.com/apache/arrow-testing/ ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-940498802
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937465189
Revision: 96823881bfd42876dfbcc577415a1870611fee25
Submitted crossbow builds: [ursacomputing/crossbow @ actions-903](https://github.com/ursacomputing/crossbow/branches/all?query=actions-903)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-903-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-903-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-943214026
Also, can you rebase from master to ensure that we get the latest CI fixes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-939888418
Revision: 3c32de96b05171c126d1cf5e08ce019eaedb244f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-910](https://github.com/ursacomputing/crossbow/branches/all?query=actions-910)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-910-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-910-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on a change in pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r724464912
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/csv/api.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/api.h>
+#include <arrow/io/api.h>
+#include <arrow/io/compressed.h>
+#include <arrow/ipc/api.h>
+#include <arrow/pretty_print.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+#include <gmock/gmock-matchers.h>
+
+#include <iostream>
+
+namespace arrow {
+
+Result<std::shared_ptr<dataset::Dataset>> MakeDataset(const std::string& filename) {
+ auto fs = std::make_shared<fs::LocalFileSystem>();
+ auto fs_dataset_options = arrow::dataset::FileSystemFactoryOptions();
+ auto format = std::make_shared<arrow::dataset::IpcFileFormat>();
+ ARROW_ASSIGN_OR_RAISE(auto dataset_factory,
+ arrow::dataset::FileSystemDatasetFactory::Make(
+ fs, {filename}, format, fs_dataset_options));
+ auto finish_options = arrow::dataset::FinishOptions();
+ finish_options.validate_fragments = false;
+
+ auto inspect_options = arrow::dataset::InspectOptions();
+ inspect_options.fragments = 1;
+ finish_options.inspect_options = inspect_options;
+ return dataset_factory->Finish(finish_options);
+}
+
+Result<std::shared_ptr<dataset::ScanOptions>> DefaultOptions(
+ std::shared_ptr<dataset::Dataset> dataset) {
+ auto scanner_builder = std::make_shared<dataset::ScannerBuilder>(std::move(dataset));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseAsync(true));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseThreads(true));
+ ARROW_ASSIGN_OR_RAISE(auto scanner, scanner_builder->Finish());
+ return scanner->options();
+}
+
+Status RunMain() {
+ dataset::internal::Initialize();
+ ARROW_ASSIGN_OR_RAISE(
+ auto line_items_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/lineitem_1.uncompressed.feather"));
+ ARROW_ASSIGN_OR_RAISE(auto line_items_scan_opts, DefaultOptions(line_items_dataset));
+ ARROW_ASSIGN_OR_RAISE(
+ auto orders_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/orders_1.uncompressed.feather"));
Review comment:
My mistake. This file contains a repro for a different issue and should not have been included. I updated PR removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937439647
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-940720773
Revision: 3d1ce0c31fdab96ef969d790fc3ac673c8749379
Submitted crossbow builds: [ursacomputing/crossbow @ actions-917](https://github.com/ursacomputing/crossbow/branches/all?query=actions-917)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-917-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-917-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-939592625
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-940499117
Revision: 1deb369bd9dccd27475ee47574ec14015f35b532
Submitted crossbow builds: [ursacomputing/crossbow @ actions-915](https://github.com/ursacomputing/crossbow/branches/all?query=actions-915)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-915-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-915-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-939592870
Revision: 679309bb681dd4e40caf5cccf608a867d97f3add
Submitted crossbow builds: [ursacomputing/crossbow @ actions-908](https://github.com/ursacomputing/crossbow/branches/all?query=actions-908)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-908-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-908-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937440090
Revision: 16ded42dddc5e654dd177707c1c47dd6d8005c0d
Submitted crossbow builds: [ursacomputing/crossbow @ actions-902](https://github.com/ursacomputing/crossbow/branches/all?query=actions-902)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-902-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-902-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937464815
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-943145948
Revision: c94bd41ed08dcbb3dd22585cf97fa536abc48262
Submitted crossbow builds: [ursacomputing/crossbow @ actions-952](https://github.com/ursacomputing/crossbow/branches/all?query=actions-952)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-952-azure-test-conda-cpp-valgrind)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-debian-11-cpp)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-fedora-33-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-952-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946364232
Benchmark runs are scheduled for baseline = 6e1293b8b492d02a31961997173128651be62b9a and contender = d9ef519f458fc5989fd15f0af49f069c34110c35. d9ef519f458fc5989fd15f0af49f069c34110c35 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/009312669c4a4a7688c47ed88407cb97...fc83bba11fcd43f08b0eb86a45d29bf9/)
[Finished :arrow_down:0.51% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/99cf4e2619b940bbb391f96070a47aff...384eee012ee04f74a64e385b10a8810b/)
[Finished :arrow_down:0.36% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/d4b37798cbe14d90b7f2636bf8e1ff02...44e3cedae529460ab28d50c1a11b9a07/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946364232
Benchmark runs are scheduled for baseline = 6e1293b8b492d02a31961997173128651be62b9a and contender = d9ef519f458fc5989fd15f0af49f069c34110c35. d9ef519f458fc5989fd15f0af49f069c34110c35 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/009312669c4a4a7688c47ed88407cb97...fc83bba11fcd43f08b0eb86a45d29bf9/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/99cf4e2619b940bbb391f96070a47aff...384eee012ee04f74a64e385b10a8810b/)
[Finished :arrow_down:0.36% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/d4b37798cbe14d90b7f2636bf8e1ff02...44e3cedae529460ab28d50c1a11b9a07/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on a change in pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
bkietz commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r731151891
##########
File path: cpp/src/arrow/compute/exec/hash_join.cc
##########
@@ -636,6 +637,47 @@ class HashJoinBasicImpl : public HashJoinImpl {
return ScanHashTable(thread_index);
}
+ void InitHasMatchIfNeeded(ThreadLocalState* local_state) {
+ if (local_state->is_has_match_initialized) {
+ return;
+ }
+ if (!hash_table_empty_) {
+ int32_t num_rows = hash_table_keys_.num_rows();
+ local_state->has_match.resize(BitUtil::BytesForBits(num_rows));
+ memset(local_state->has_match.data(), 0, BitUtil::BytesForBits(num_rows));
+ }
+ local_state->is_has_match_initialized = true;
+ }
+
+ void MergeHasMatch() {
+ if (hash_table_empty_) {
+ return;
+ }
+
+ int32_t num_rows = hash_table_keys_.num_rows();
+ has_match_.resize(BitUtil::BytesForBits(num_rows));
+ memset(has_match_.data(), 0, BitUtil::BytesForBits(num_rows));
+
+ for (size_t tid = 0; tid < local_states_.size(); ++tid) {
+ if (!local_states_[tid].is_initialized) {
+ continue;
+ }
+ if (!local_states_[tid].is_has_match_initialized) {
+ continue;
+ }
+ uint8_t* dst8 = has_match_.data();
+ const uint8_t* src8 = local_states_[tid].has_match.data();
+ uint64_t* dst64 = reinterpret_cast<uint64_t*>(dst8);
+ const uint64_t* src64 = reinterpret_cast<const uint64_t*>(src8);
+ for (int32_t i = 0; i < num_rows / 64; ++i) {
+ dst64[i] |= src64[i];
+ }
+ for (int32_t i = num_rows / 64 * 8; i < BitUtil::CeilDiv(num_rows, 8); ++i) {
+ dst8[i] |= src8[i];
+ }
Review comment:
```suggestion
arrow::internal::BitmapOr(has_match_.data(), 0, local_states_[tid].has_match.data(),
0, num_rows, has_match_.data(), 0);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946165042
@github-actions crossbow submit -g cpp -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946364232
Benchmark runs are scheduled for baseline = 6e1293b8b492d02a31961997173128651be62b9a and contender = d9ef519f458fc5989fd15f0af49f069c34110c35. d9ef519f458fc5989fd15f0af49f069c34110c35 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/009312669c4a4a7688c47ed88407cb97...fc83bba11fcd43f08b0eb86a45d29bf9/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/99cf4e2619b940bbb391f96070a47aff...384eee012ee04f74a64e385b10a8810b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/d4b37798cbe14d90b7f2636bf8e1ff02...44e3cedae529460ab28d50c1a11b9a07/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937439660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-943141972
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-938079843
https://issues.apache.org/jira/browse/ARROW-14211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11350: ARROW-14197: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-937439660
https://issues.apache.org/jira/browse/ARROW-14197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-939882332
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
kou closed pull request #11350:
URL: https://github.com/apache/arrow/pull/11350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on a change in pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r724527144
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
Review comment:
I removed tpch_test.cc - it was added by mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on a change in pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r724464912
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/csv/api.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/api.h>
+#include <arrow/io/api.h>
+#include <arrow/io/compressed.h>
+#include <arrow/ipc/api.h>
+#include <arrow/pretty_print.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+#include <gmock/gmock-matchers.h>
+
+#include <iostream>
+
+namespace arrow {
+
+Result<std::shared_ptr<dataset::Dataset>> MakeDataset(const std::string& filename) {
+ auto fs = std::make_shared<fs::LocalFileSystem>();
+ auto fs_dataset_options = arrow::dataset::FileSystemFactoryOptions();
+ auto format = std::make_shared<arrow::dataset::IpcFileFormat>();
+ ARROW_ASSIGN_OR_RAISE(auto dataset_factory,
+ arrow::dataset::FileSystemDatasetFactory::Make(
+ fs, {filename}, format, fs_dataset_options));
+ auto finish_options = arrow::dataset::FinishOptions();
+ finish_options.validate_fragments = false;
+
+ auto inspect_options = arrow::dataset::InspectOptions();
+ inspect_options.fragments = 1;
+ finish_options.inspect_options = inspect_options;
+ return dataset_factory->Finish(finish_options);
+}
+
+Result<std::shared_ptr<dataset::ScanOptions>> DefaultOptions(
+ std::shared_ptr<dataset::Dataset> dataset) {
+ auto scanner_builder = std::make_shared<dataset::ScannerBuilder>(std::move(dataset));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseAsync(true));
+ ARROW_RETURN_NOT_OK(scanner_builder->UseThreads(true));
+ ARROW_ASSIGN_OR_RAISE(auto scanner, scanner_builder->Finish());
+ return scanner->options();
+}
+
+Status RunMain() {
+ dataset::internal::Initialize();
+ ARROW_ASSIGN_OR_RAISE(
+ auto line_items_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/lineitem_1.uncompressed.feather"));
+ ARROW_ASSIGN_OR_RAISE(auto line_items_scan_opts, DefaultOptions(line_items_dataset));
+ ARROW_ASSIGN_OR_RAISE(
+ auto orders_dataset,
+ MakeDataset("~/tpch_repro/ARROW-14197-input/orders_1.uncompressed.feather"));
Review comment:
My mistake. This file contains a repro for a different issue and should not have been included. I updated PR removing it.
##########
File path: cpp/src/arrow/compute/exec/tpch_test.cc
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
Review comment:
I removed tpch_test.cc - it was added by mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-943142892
@michalursa Is this ready for review? Ideally we would like to ship this fix in 6.0.0, which will hopefully be cut in a few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on a change in pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on a change in pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#discussion_r730199563
##########
File path: cpp/src/arrow/compute/exec/hash_join.cc
##########
@@ -150,23 +154,26 @@ class HashJoinBasicImpl : public HashJoinImpl {
int num_cols = schema_mgr_->proj_maps[side].num_cols(projection_handle);
projected.values.resize(num_cols);
- const int* to_input =
+ auto to_input =
schema_mgr_->proj_maps[side].map(projection_handle, HashJoinProjection::INPUT);
for (int icol = 0; icol < num_cols; ++icol) {
- projected.values[icol] = batch.values[to_input[icol]];
+ projected.values[icol] = batch.values[to_input.get(icol)];
}
return encoder->EncodeAndAppend(projected);
}
- void ProbeBatch_Lookup(const RowEncoder& exec_batch_keys,
+ void ProbeBatch_Lookup(ThreadLocalState& local_state, const RowEncoder& exec_batch_keys,
Review comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] michalursa commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
michalursa commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-940712624
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946363711
OK.
I merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #11350: ARROW-14211: [C++][Compute] Fixing thread sanitizer problems in hash join node
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11350:
URL: https://github.com/apache/arrow/pull/11350#issuecomment-946364232
Benchmark runs are scheduled for baseline = 6e1293b8b492d02a31961997173128651be62b9a and contender = d9ef519f458fc5989fd15f0af49f069c34110c35. d9ef519f458fc5989fd15f0af49f069c34110c35 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/009312669c4a4a7688c47ed88407cb97...fc83bba11fcd43f08b0eb86a45d29bf9/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/99cf4e2619b940bbb391f96070a47aff...384eee012ee04f74a64e385b10a8810b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/d4b37798cbe14d90b7f2636bf8e1ff02...44e3cedae529460ab28d50c1a11b9a07/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org