You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/10 02:13:29 UTC
[GitHub] [arrow] cyb70289 opened a new pull request #10283: ARROW-12533: [C++] Add random real distribution function
cyb70289 opened a new pull request #10283:
URL: https://github.com/apache/arrow/pull/10283
Clang with gnu libstdc++ produces code very slow in generating random
real numbers on Arm64.
This patch implements three random utilities based on clang libc++:
- std::generate_canonical
- std::random_real_distribution
- std::bernoulli_distribution
It brings ~100x speedup on Arm64 and ~8x on x86_64 in generating
random reals when build arrow with clang + gnu libstdc++.
No influence to gcc + libstdc++, or clang + libc++.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #10283:
URL: https://github.com/apache/arrow/pull/10283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-838127459
Thank you @cyb70289 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 edited a comment on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 edited a comment on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836076518
Tested on Xeon Gold 5218. Clang10, libstdc++-9.
Before: **5.4s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 550 us 550 us 1279 bytes_per_second=1.7744G/s null_percent=0.01 size=1048.58k
real 0m5.445s
user 0m5.417s
sys 0m0.008s
```
After: **1.2s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 551 us 551 us 1267 bytes_per_second=1.7723G/s null_percent=0.01 size=1048.58k
real 0m1.197s
user 0m1.180s
sys 0m0.000s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on a change in pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#discussion_r629807641
##########
File path: cpp/src/arrow/testing/uniform_real.h
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Based on clang libc++: https://github.com/llvm/llvm-project/tree/main/libcxx
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836147132
Timed running all arrow-compute-*-benchmark tests on x86, clang. Total time drops from 26 to 18 min.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 edited a comment on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 edited a comment on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836076149
**NOTE:** Below benchmark results are the same, but the total run time differs much. The saved time is from test preparation (generating random bools by calling bernoulli_distribution), not the test itself.
Test on Neoverse N1. Clang-10, libstdc++-9.
Before: **59.9s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 637 us 637 us 1097 bytes_per_second=1.53351G/s null_percent=0.01 size=1048.58k
real 0m59.902s
user 0m59.880s
sys 0m0.020s
```
After: **1.2s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 637 us 637 us 1098 bytes_per_second=1.53253G/s null_percent=0.01 size=1048.58k
real 0m1.210s
user 0m1.202s
sys 0m0.008s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836070781
https://issues.apache.org/jira/browse/ARROW-12533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836148484
Travis CI passed at https://travis-ci.org/github/cyb70289/arrow/builds/770391382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836076518
Tested on Xeon Gold 5218.
Before: **5.4s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 550 us 550 us 1279 bytes_per_second=1.7744G/s null_percent=0.01 size=1048.58k
real 0m5.445s
user 0m5.417s
sys 0m0.008s
```
After: **1.2s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 551 us 551 us 1267 bytes_per_second=1.7723G/s null_percent=0.01 size=1048.58k
real 0m1.197s
user 0m1.180s
sys 0m0.000s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#discussion_r629415113
##########
File path: cpp/src/arrow/testing/uniform_real.h
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Based on clang libc++: https://github.com/llvm/llvm-project/tree/main/libcxx
Review comment:
Can you just add a comment explaining the motivation for this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #10283: ARROW-12533: [C++] Add random real distribution function
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #10283:
URL: https://github.com/apache/arrow/pull/10283#issuecomment-836076149
**NOTE:** Below benchmark results are the same, but the total run time differs much. The saved time is from test preparation (generating random bools by calling bernoulli_distribution), not the test itself.
Test on Neoverse N1, clang10.
Before: **59.9s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 637 us 637 us 1097 bytes_per_second=1.53351G/s null_percent=0.01 size=1048.58k
real 0m59.902s
user 0m59.880s
sys 0m0.020s
```
After: **1.2s** total run time
```
$ time release/arrow-compute-aggregate-benchmark --benchmark_filter="ModeKernelNarrow<BooleanType>/1048576/10000"
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
ModeKernelNarrow<BooleanType>/1048576/10000 637 us 637 us 1098 bytes_per_second=1.53253G/s null_percent=0.01 size=1048.58k
real 0m1.210s
user 0m1.202s
sys 0m0.008s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org