You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zanmato1984 (via GitHub)" <gi...@apache.org> on 2024/01/25 16:47:36 UTC
[PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
zanmato1984 opened a new pull request, #39800:
URL: https://github.com/apache/arrow/pull/39800
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
Issue #39778 seems caused by a careless (but hard to spot) bug in key hash avx2.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
Fix the careless bug.
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
UT included.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
No.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39800:
URL: https://github.com/apache/arrow/pull/39800#issuecomment-1912084234
Revision: 5fea9bbd934561f4359c9f4df2a3a7e049f0a1b1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-cfeb62c110](https://github.com/ursacomputing/crossbow/branches/all?query=actions-cfeb62c110)
|Task|Status|
|----|------|
|test-alpine-linux-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791332/job/20901309475)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791501/job/20901309634)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791307/job/20901308866)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/runs/20901310989)|
|test-cuda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791308/job/20901308863)|
|test-debian-11-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791469/job/20901309552)|
|test-debian-11-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791573/job/20901309874)|
|test-fedora-38-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791552/job/20901309939)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791585/job/20901310091)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791390/job/20901309438)|
|test-ubuntu-20.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791339/job/20901309154)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791305/job/20901308873)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791359/job/20901309211)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791509/job/20901309814)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/7668791329/job/20901309083)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466737628
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
I guess we better do so. Let me update them in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466748740
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Or `key_width`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466733100
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Oh, darn. Could we take the opportunity to update signature and call sites? Call it `row_length` for example.
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Oh, darn. Could we take the opportunity to update signatures and call sites? Call it `row_length` for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #39800:
URL: https://github.com/apache/arrow/pull/39800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #39800:
URL: https://github.com/apache/arrow/pull/39800#issuecomment-1912493438
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 13b22346d36b9952df5c988c9425b9e5bc4f09c4.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/20911196776) has more details. It also includes information about 202 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #39800:
URL: https://github.com/apache/arrow/pull/39800#issuecomment-1910595826
cc @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #39800:
URL: https://github.com/apache/arrow/pull/39800#issuecomment-1912081099
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466730093
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
I guess you were thinking `length` is some kind of "total length" of all rows? But it actually represents the width of the fixed length type being hashed. See its call site:
https://github.com/apache/arrow/blob/9c0a7617d953a74abaec5c14318bc2f46d0d176a/cpp/src/arrow/compute/key_hash.cc#L431C69-L431C78
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466753723
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
> Actually, perhaps `key_length`?
Deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466741459
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Just to confirm before I make the change, how do you think the name `fixed_length`? I think it might be more specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466692173
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Is this right? I don't understand why we would divide a constant-time stripe size by a runtime length.
Don't you instead simply want something like:
```c++
uint32_t num_rows_to_process = length / kStripeSize * kStripeSize;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #39800:
URL: https://github.com/apache/arrow/pull/39800#issuecomment-1912504639
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 13b22346d36b9952df5c988c9425b9e5bc4f09c4.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/20911487337) has more details. It also includes information about 202 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466784649
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Done.
Changed every misleading `length` to `key_length`. Additionally changed some other names to make naming slightly more consistent between header and cpp, function counterparts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466761561
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Both would be fine with me :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39778: [C++] Fix tail-byte access cross buffer boundary in key hash avx2 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39800:
URL: https://github.com/apache/arrow/pull/39800#discussion_r1466746463
##########
cpp/src/arrow/compute/key_hash_avx2.cc:
##########
@@ -190,7 +190,7 @@ uint32_t Hashing32::HashFixedLenImp_avx2(uint32_t num_rows, uint64_t length,
// Do not process rows that could read past the end of the buffer using 16
// byte loads. Round down number of rows to process to multiple of 2.
//
- uint64_t num_rows_to_skip = bit_util::CeilDiv(length, kStripeSize);
+ uint64_t num_rows_to_skip = bit_util::CeilDiv(kStripeSize, length);
Review Comment:
Actually, perhaps `key_length`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org