You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2024/03/05 09:12:48 UTC
[PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
AlenkaF opened a new pull request, #40358:
URL: https://github.com/apache/arrow/pull/40358
### Rationale for this change
We should add benchmarks to be sure not to cause regressions while working on additional implementations of `RecordBatch::ToTensor` and `Table::ToTensor`.
### What changes are included in this PR?
New `cpp/src/arrow/to_tensor_benchmark.cc file`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1537909877
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t num_cols = state.range(1);
+ const int64_t num_rows = state.range(0) / num_cols / sizeof(CType);
+ arrow::random::RandomArrayGenerator gen_{42};
+
+ std::vector<std::shared_ptr<Field>> fields = {};
+ std::vector<std::shared_ptr<Array>> columns = {};
+
+ for (int64_t i = 0; i < num_cols; ++i) {
+ fields.push_back(field("f" + std::to_string(i), ty));
+ columns.push_back(gen_.ArrayOf(ty, num_rows));
+ }
+ auto schema = std::make_shared<Schema>(std::move(fields));
+ auto batch = RecordBatch::Make(schema, num_rows, columns);
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto tensor, batch->ToTensor());
+ }
+ state.SetItemsProcessed(state.iterations() * num_rows * num_cols);
+ state.SetBytesProcessed(state.iterations() * ty->byte_width() * num_rows * num_cols);
+}
+
+void SetArgs(benchmark::internal::Benchmark* bench) {
+ for (int64_t size : {kL1Size, kL2Size}) {
+ for (int64_t number_of_columns : {3, 30, 300}) {
+ bench->Args({size, number_of_columns});
Review Comment:
Just a nit here, can we also give the parameters names to make the output more readable?
For example:
```suggestion
bench->ArgNames({"size", "num_columns"});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-1980353727
This was the result output:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-ahcnq1ah/WORKSPACE/build/release/arrow-to-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 17.32, 18.72, 16.18
----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
----------------------------------------------------------------------------------------
RecordBatchUniformTypesSimple 624 ns 624 ns 1125492 bytes_per_second=1.29039Gi/s items_per_second=43.2982M/s
```
WIll use `RandomArrayGenerator` to generate more data and add the result here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1532123779
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size;
Review Comment:
`bytes_per_second=143.748Mi/s items_per_second=15.073G/s` doesn't make sense, does it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-1994442166
Current output when running `archery benchmark diff --benchmark-filter=BatchToTensorSimple`:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-e8lvkw1g/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 27.50, 28.87, 23.74
-----------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------------
BatchToTensorSimple<UInt8Type>/65536/10000 4121 us 4107 us 171 bytes_per_second=15.217Mi/s items_per_second=12.765G/s null_percent=0.01 size=65.536k
BatchToTensorSimple<UInt8Type>/65536/100 4273 us 4219 us 170 bytes_per_second=14.8143Mi/s items_per_second=12.4271G/s null_percent=1 size=65.536k
BatchToTensorSimple<UInt8Type>/65536/10 4019 us 4003 us 173 bytes_per_second=15.6149Mi/s items_per_second=13.0988G/s null_percent=10 size=65.536k
BatchToTensorSimple<UInt8Type>/65536/2 4100 us 4083 us 136 bytes_per_second=15.3084Mi/s items_per_second=12.8416G/s null_percent=50 size=65.536k
BatchToTensorSimple<UInt8Type>/65536/1 3972 us 3894 us 178 bytes_per_second=16.0516Mi/s items_per_second=13.465G/s null_percent=100 size=65.536k
BatchToTensorSimple<UInt8Type>/65536/0 3953 us 3927 us 178 bytes_per_second=15.9142Mi/s items_per_second=13.3498G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt8Type>/4194304/10000 15398661 us 1947088 us 1 bytes_per_second=2.05435Mi/s items_per_second=1.72331G/s null_percent=0.01 size=4.1943M
.
.
.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1533872400
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size;
Review Comment:
I guess not. What I understand, at least, is that the number for `items_per_second` should be approx `bytes_per_second` divided by the size of the type. Joris advised me what I could try to debug this but I am not finding anything I could grasp.
I am not really sure if it makes a difference if I only use `state.SetBytesProcessed` without `state.SetItemsProcessed`. It also looks OK if I just leave both of them out:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-vd706e0e/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 15.15, 15.26, 13.19
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------
BatchToTensorSimple<UInt8Type>/65536 429847 ns 429439 ns 1582 bytes_per_second=145.539Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt8Type>/4194304 56283753 ns 44952231 ns 13 bytes_per_second=88.9833Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt16Type>/65536 470726 ns 462170 ns 1607 bytes_per_second=135.232Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt16Type>/4194304 44393589 ns 37141214 ns 14 bytes_per_second=107.697Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt32Type>/65536 440997 ns 439951 ns 1260 bytes_per_second=142.061Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt32Type>/4194304 43955912 ns 36447556 ns 18 bytes_per_second=109.747Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt64Type>/65536 432952 ns 431213 ns 1369 bytes_per_second=144.94Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt64Type>/4194304 40377762 ns 36827529 ns 17 bytes_per_second=108.614Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int8Type>/65536 583566 ns 561105 ns 1667 bytes_per_second=111.387Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<Int8Type>/4194304 69477871 ns 51189900 ns 10 bytes_per_second=78.1404Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int16Type>/65536 466828 ns 460938 ns 1379 bytes_per_second=135.593Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<Int16Type>/4194304 53699115 ns 43646833 ns 12 bytes_per_second=91.6447Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int32Type>/65536 510174 ns 489199 ns 1380 bytes_per_second=127.76Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<Int32Type>/4194304 59453215 ns 43936000 ns 13 bytes_per_second=91.0415Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int64Type>/65536 449931 ns 446273 ns 1581 bytes_per_second=140.049Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<Int64Type>/4194304 44797259 ns 38353000 ns 19 bytes_per_second=104.294Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<HalfFloatType>/65536 501073 ns 470337 ns 1660 bytes_per_second=132.884Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<HalfFloatType>/4194304 57234822 ns 40693467 ns 15 bytes_per_second=98.2959Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<FloatType>/65536 420881 ns 419577 ns 1389 bytes_per_second=148.96Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<FloatType>/4194304 41806079 ns 37133778 ns 18 bytes_per_second=107.719Mi/s null_percent=0 size=4.1943M
BatchToTensorSimple<DoubleType>/65536 424610 ns 423430 ns 1346 bytes_per_second=147.604Mi/s null_percent=0 size=65.536k
BatchToTensorSimple<DoubleType>/4194304 37983824 ns 35989222 ns 18 bytes_per_second=111.144Mi/s null_percent=0 size=4.1943M
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-1979175710
Can you show the result of running them? And we might want to use some more data to get a more reliable result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1533904795
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size;
Review Comment:
It might be caused by using `RegressionArgs`, which also calls `SetBytesProcessed` in its destructor (now, if that's the case, then we have some other benchmarks reporting the wrong number as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-2020577498
After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit fc87fd75d6602562e64abf8744890332e35f979e.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/23107943790) has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-2012348750
Looking better after your last suggestions Joris:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-w9c6kiee/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 16.45, 17.62, 11.63
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------
BatchToTensorSimple<UInt8Type>/65536 458536 ns 458333 ns 1504 bytes_per_second=13.3168Gi/s items_per_second=14.2988G/s
BatchToTensorSimple<UInt8Type>/4194304 51650596 ns 40859385 ns 13 bytes_per_second=9.56023Gi/s items_per_second=10.2652G/s
BatchToTensorSimple<UInt16Type>/65536 443072 ns 441767 ns 1327 bytes_per_second=13.8161Gi/s items_per_second=7.41748G/s
BatchToTensorSimple<UInt16Type>/4194304 37997653 ns 36128500 ns 18 bytes_per_second=10.8121Gi/s items_per_second=5.8047G/s
BatchToTensorSimple<UInt32Type>/65536 556504 ns 525753 ns 1625 bytes_per_second=11.6091Gi/s items_per_second=3.11629G/s
BatchToTensorSimple<UInt32Type>/4194304 47726554 ns 38831059 ns 17 bytes_per_second=10.0596Gi/s items_per_second=2.70035G/s
BatchToTensorSimple<UInt64Type>/65536 543929 ns 510296 ns 1071 bytes_per_second=11.9607Gi/s items_per_second=1.60534G/s
BatchToTensorSimple<UInt64Type>/4194304 104417887 ns 55937176 ns 17 bytes_per_second=6.98328Gi/s items_per_second=937.28M/s
BatchToTensorSimple<Int8Type>/65536 542291 ns 530495 ns 1000 bytes_per_second=11.5053Gi/s items_per_second=12.3537G/s
BatchToTensorSimple<Int8Type>/4194304 55069580 ns 44818231 ns 13 bytes_per_second=8.71576Gi/s items_per_second=9.35848G/s
BatchToTensorSimple<Int16Type>/65536 472947 ns 466738 ns 1604 bytes_per_second=13.077Gi/s items_per_second=7.02065G/s
BatchToTensorSimple<Int16Type>/4194304 45937775 ns 40318200 ns 15 bytes_per_second=9.68855Gi/s items_per_second=5.2015G/s
BatchToTensorSimple<Int32Type>/65536 439955 ns 438705 ns 1351 bytes_per_second=13.9126Gi/s items_per_second=3.73463G/s
BatchToTensorSimple<Int32Type>/4194304 38181667 ns 36099833 ns 18 bytes_per_second=10.8207Gi/s items_per_second=2.90466G/s
BatchToTensorSimple<Int64Type>/65536 440425 ns 439585 ns 1583 bytes_per_second=13.8847Gi/s items_per_second=1.86358G/s
BatchToTensorSimple<Int64Type>/4194304 51548936 ns 39940333 ns 15 bytes_per_second=9.78021Gi/s items_per_second=1.31268G/s
BatchToTensorSimple<HalfFloatType>/65536 435417 ns 434107 ns 1526 bytes_per_second=14.0599Gi/s items_per_second=7.54836G/s
BatchToTensorSimple<HalfFloatType>/4194304 48649122 ns 38652385 ns 13 bytes_per_second=10.1061Gi/s items_per_second=5.42567G/s
BatchToTensorSimple<FloatType>/65536 432115 ns 430647 ns 1522 bytes_per_second=14.1729Gi/s items_per_second=3.80451G/s
BatchToTensorSimple<FloatType>/4194304 42923344 ns 38628000 ns 16 bytes_per_second=10.1125Gi/s items_per_second=2.71455G/s
BatchToTensorSimple<DoubleType>/65536 442113 ns 441402 ns 1304 bytes_per_second=13.8276Gi/s items_per_second=1.85591G/s
BatchToTensorSimple<DoubleType>/4194304 60867021 ns 44292875 ns 16 bytes_per_second=8.81914Gi/s items_per_second=1.18368G/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-2012374041
Thanks for this @AlenkaF . I have two general suggestions here:
1) given that the types are purely physical here (i.e. float32 should use the same conversion code as int32 and uint32), we don't need to benchmark all numeric data types, we can limit ourselves to four integer types: int8, int16, int32, int64
2) on the other hand, it would be nice to exercise different numbers of columns, because that could affect conversion performance: for example 3, 30, 300?
Does it make sense @AlenkaF @jorisvandenbossche ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1515763364
##########
cpp/src/arrow/to_tensor_benchmark.cc:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+ auto f0 = field("f0", ty);
+ auto f1 = field("f1", ty);
+ auto f2 = field("f2", ty);
+
+ std::vector<std::shared_ptr<Field>> fields = {f0, f1, f2};
+ auto schema = ::arrow::schema(fields);
+
+ constexpr int kNumRows = 500;
Review Comment:
I think you can go with something much bigger, like 100k
##########
cpp/src/arrow/to_tensor_benchmark.cc:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+ auto f0 = field("f0", ty);
+ auto f1 = field("f1", ty);
+ auto f2 = field("f2", ty);
+
+ std::vector<std::shared_ptr<Field>> fields = {f0, f1, f2};
+ auto schema = ::arrow::schema(fields);
+
+ constexpr int kNumRows = 500;
Review Comment:
And maybe also more columns. We could also define a total number of elements, and then test with multiple number of columns (eg 10 and 100) and calculate the the number of rows from that
##########
cpp/src/arrow/to_tensor_benchmark.cc:
##########
Review Comment:
To keep with naming conventions, maybe call this `tensor_benchmark.cc`? (so that it's next to a `tensor.cc`, which will be correct when we move the code there)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1532133438
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size / sizeof(CType);
+ arrow::random::RandomArrayGenerator gen_{42};
+
+ std::vector<std::shared_ptr<Field>> fields = {};
+ std::vector<std::shared_ptr<Array>> columns = {};
+
+ for (int i = 0; i < 100; ++i) {
+ fields.push_back(field("f" + std::to_string(i), ty));
+ columns.push_back(gen_.ArrayOf(ty, kNumRows));
+ }
+ auto schema = std::make_shared<Schema>(std::move(fields));
+ auto batch = RecordBatch::Make(schema, kNumRows, columns);
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto tensor, batch->ToTensor());
+ }
+ state.SetItemsProcessed(state.iterations() * kNumRows * batch->num_columns());
+ state.SetBytesProcessed(state.iterations() * ty->bit_width() * kNumRows *
Review Comment:
```suggestion
state.SetBytesProcessed(state.iterations() * ty->byte_width() * kNumRows *
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-2012426569
It does! Will update 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1526056583
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size;
Review Comment:
I would maybe still do the ` / 8` ( or division by `sizeof(CType)`), because the reported "Time" of some of benchmarks is still in the > second range
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1538012590
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t num_cols = state.range(1);
+ const int64_t num_rows = state.range(0) / num_cols / sizeof(CType);
+ arrow::random::RandomArrayGenerator gen_{42};
+
+ std::vector<std::shared_ptr<Field>> fields = {};
+ std::vector<std::shared_ptr<Array>> columns = {};
+
+ for (int64_t i = 0; i < num_cols; ++i) {
+ fields.push_back(field("f" + std::to_string(i), ty));
+ columns.push_back(gen_.ArrayOf(ty, num_rows));
+ }
+ auto schema = std::make_shared<Schema>(std::move(fields));
+ auto batch = RecordBatch::Make(schema, num_rows, columns);
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto tensor, batch->ToTensor());
+ }
+ state.SetItemsProcessed(state.iterations() * num_rows * num_cols);
+ state.SetBytesProcessed(state.iterations() * ty->byte_width() * num_rows * num_cols);
+}
+
+void SetArgs(benchmark::internal::Benchmark* bench) {
+ for (int64_t size : {kL1Size, kL2Size}) {
+ for (int64_t number_of_columns : {3, 30, 300}) {
+ bench->Args({size, number_of_columns});
Review Comment:
Sure! Just to be clear, do we need `bench->Args(...)` and `bench->ArgNames(...)` together or is one (`bench->ArgNames(...)`) enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1523336982
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size * 8;
Review Comment:
If you are doing `* 8 ` for the reason of converting bytes to number of elements, this should actually be `/` (and the 8 should also depend on the actual type).
But to simplify (and since you are setting `SetItemsProcessed`/`SetBytesProcessed` manually anyway), I think you can also just see `args.size` as the number of elements in the array
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size * 8;
+ arrow::random::RandomArrayGenerator gen_{42};
+
+ std::vector<std::shared_ptr<Field>> fields = {};
+ std::vector<std::shared_ptr<Array>> columns = {};
+
+ for (int i = 0; i < 100; ++i) {
+ fields.push_back(field("f" + std::to_string(i), ty));
+ columns.push_back(gen_.ArrayOf(ty, kNumRows));
+ }
+ auto schema = std::make_shared<Schema>(std::move(fields));
+ auto batch = RecordBatch::Make(schema, kNumRows, columns);
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto tensor, batch->ToTensor());
+ }
+ state.SetItemsProcessed(state.iterations() * kNumRows * batch->num_columns());
+ state.SetBytesProcessed(state.iterations() * ty->bit_width() * kNumRows *
+ batch->num_columns());
+}
+
+void SetArgs(benchmark::internal::Benchmark* bench) {
+ BenchmarkSetArgsWithSizes(bench, {kL1Size, kL2Size});
Review Comment:
You might want to set the args manually here, because with `BenchmarkSetArgsWithSizes` it will also set the `null_proportion`, but in the current benchmark you are ignoring that parameter (and therefore repeating the benchmark multiple times with the same setup, as you can also see in the output you printed)
Something like
```
for (int64_t size : {kL1Size, kL2Size}) {
bench->Arg(size);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-1997567607
Output from running the benchmarks on the latest commit:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-y9o8zv4d/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 20.67, 17.39, 10.95
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------
BatchToTensorSimple<UInt8Type>/65536 443099 ns 442863 ns 1580 bytes_per_second=141.127Mi/s items_per_second=14.7983G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt8Type>/4194304 38391076 ns 35795222 ns 18 bytes_per_second=111.747Mi/s items_per_second=11.7175G/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt16Type>/65536 882040 ns 881129 ns 747 bytes_per_second=70.9318Mi/s items_per_second=7.43773G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt16Type>/4194304 118462838 ns 81059222 ns 9 bytes_per_second=49.3466Mi/s items_per_second=5.17437G/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt32Type>/65536 1937139 ns 1933673 ns 361 bytes_per_second=32.3219Mi/s items_per_second=3.3892G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt32Type>/4194304 1271556625 ns 651396000 ns 1 bytes_per_second=6.14066Mi/s items_per_second=643.895M/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt64Type>/65536 4440503 ns 4344614 ns 166 bytes_per_second=14.3856Mi/s items_per_second=1.50844G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt64Type>/4194304 1.1486e+10 ns 1742537000 ns 1 bytes_per_second=2.2955Mi/s items_per_second=240.701M/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int8Type>/65536 415187 ns 410957 ns 1710 bytes_per_second=152.084Mi/s items_per_second=15.9472G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int8Type>/4194304 34241740 ns 33962150 ns 20 bytes_per_second=117.778Mi/s items_per_second=12.3499G/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int16Type>/65536 812298 ns 810349 ns 917 bytes_per_second=77.1273Mi/s items_per_second=8.08738G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int16Type>/4194304 75301182 ns 70352375 ns 8 bytes_per_second=56.8566Mi/s items_per_second=5.96185G/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int32Type>/65536 2033466 ns 2026663 ns 329 bytes_per_second=30.8389Mi/s items_per_second=3.23369G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int32Type>/4194304 1233238541 ns 562396000 ns 1 bytes_per_second=7.11243Mi/s items_per_second=745.792M/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int64Type>/65536 3969188 ns 3959770 ns 178 bytes_per_second=15.7837Mi/s items_per_second=1.65505G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int64Type>/4194304 1.5188e+10 ns 1823171000 ns 1 bytes_per_second=2.19398Mi/s items_per_second=230.055M/s null_percent=0 size=4.1943M
BatchToTensorSimple<HalfFloatType>/65536 899771 ns 888509 ns 749 bytes_per_second=70.3426Mi/s items_per_second=7.37595G/s null_percent=0 size=65.536k
BatchToTensorSimple<HalfFloatType>/4194304 71104797 ns 69327375 ns 8 bytes_per_second=57.6973Mi/s items_per_second=6.05G/s null_percent=0 size=4.1943M
BatchToTensorSimple<FloatType>/65536 2025175 ns 2021084 ns 347 bytes_per_second=30.924Mi/s items_per_second=3.24262G/s null_percent=0 size=65.536k
BatchToTensorSimple<FloatType>/4194304 1087905188 ns 395840500 ns 2 bytes_per_second=10.1051Mi/s items_per_second=1.05959G/s null_percent=0 size=4.1943M
BatchToTensorSimple<DoubleType>/65536 4118269 ns 4089947 ns 170 bytes_per_second=15.2814Mi/s items_per_second=1.60237G/s null_percent=0 size=65.536k
BatchToTensorSimple<DoubleType>/4194304 9901101750 ns 1684713000 ns 1 bytes_per_second=2.37429Mi/s items_per_second=248.963M/s null_percent=0 size=4.1943M
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1515779406
##########
cpp/src/arrow/to_tensor_benchmark.cc:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+ auto f0 = field("f0", ty);
+ auto f1 = field("f1", ty);
+ auto f2 = field("f2", ty);
+
+ std::vector<std::shared_ptr<Field>> fields = {f0, f1, f2};
+ auto schema = ::arrow::schema(fields);
+
+ constexpr int kNumRows = 500;
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1526226212
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size;
Review Comment:
New result, dividing size by `sizeof(CType)`:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-wyoew3d4/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 19.13, 18.37, 12.07
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------
BatchToTensorSimple<UInt8Type>/65536 440965 ns 434790 ns 1628 bytes_per_second=143.748Mi/s items_per_second=15.073G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt8Type>/4194304 52116387 ns 39301000 ns 18 bytes_per_second=101.779Mi/s items_per_second=10.6723G/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt16Type>/65536 422252 ns 421368 ns 1663 bytes_per_second=148.326Mi/s items_per_second=7.77658G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt16Type>/4194304 39602325 ns 36205053 ns 19 bytes_per_second=110.482Mi/s items_per_second=5.79243G/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt32Type>/65536 411546 ns 411012 ns 1696 bytes_per_second=152.064Mi/s items_per_second=3.98625G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt32Type>/4194304 37668923 ns 35941842 ns 19 bytes_per_second=111.291Mi/s items_per_second=2.91742G/s null_percent=0 size=4.1943M
BatchToTensorSimple<UInt64Type>/65536 409912 ns 409266 ns 1772 bytes_per_second=152.712Mi/s items_per_second=2.00163G/s null_percent=0 size=65.536k
BatchToTensorSimple<UInt64Type>/4194304 40266224 ns 36517789 ns 19 bytes_per_second=109.536Mi/s items_per_second=1.43571G/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int8Type>/65536 404307 ns 403876 ns 1709 bytes_per_second=154.75Mi/s items_per_second=16.2268G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int8Type>/4194304 37406713 ns 35309316 ns 19 bytes_per_second=113.285Mi/s items_per_second=11.8787G/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int16Type>/65536 414663 ns 414136 ns 1649 bytes_per_second=150.916Mi/s items_per_second=7.91237G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int16Type>/4194304 37432355 ns 35457526 ns 19 bytes_per_second=112.811Mi/s items_per_second=5.91455G/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int32Type>/65536 413986 ns 413420 ns 1706 bytes_per_second=151.178Mi/s items_per_second=3.96304G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int32Type>/4194304 47971980 ns 37791471 ns 17 bytes_per_second=105.844Mi/s items_per_second=2.77464G/s null_percent=0 size=4.1943M
BatchToTensorSimple<Int64Type>/65536 415919 ns 415559 ns 1691 bytes_per_second=150.4Mi/s items_per_second=1.97132G/s null_percent=0 size=65.536k
BatchToTensorSimple<Int64Type>/4194304 36665862 ns 35319650 ns 20 bytes_per_second=113.251Mi/s items_per_second=1.48441G/s null_percent=0 size=4.1943M
BatchToTensorSimple<HalfFloatType>/65536 422161 ns 421677 ns 1685 bytes_per_second=148.218Mi/s items_per_second=7.77088G/s null_percent=0 size=65.536k
BatchToTensorSimple<HalfFloatType>/4194304 35648150 ns 34911650 ns 20 bytes_per_second=114.575Mi/s items_per_second=6.00703G/s null_percent=0 size=4.1943M
BatchToTensorSimple<FloatType>/65536 407051 ns 406626 ns 1702 bytes_per_second=153.704Mi/s items_per_second=4.02925G/s null_percent=0 size=65.536k
BatchToTensorSimple<FloatType>/4194304 35324888 ns 34521250 ns 20 bytes_per_second=115.871Mi/s items_per_second=3.03748G/s null_percent=0 size=4.1943M
BatchToTensorSimple<DoubleType>/65536 411345 ns 410348 ns 1740 bytes_per_second=152.31Mi/s items_per_second=1.99635G/s null_percent=0 size=65.536k
BatchToTensorSimple<DoubleType>/4194304 36834741 ns 35409211 ns 19 bytes_per_second=112.965Mi/s items_per_second=1.48065G/s null_percent=0 size=4.1943M
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #40358:
URL: https://github.com/apache/arrow/pull/40358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-2016713144
Latest output:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-1wpfanyn/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 31.29, 25.96, 16.82
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------
BatchToTensorSimple<Int8Type>/65536/3 2441 ns 2440 ns 340594 bytes_per_second=25.0118Gi/s items_per_second=26.8563G/s
BatchToTensorSimple<Int8Type>/65536/30 3158 ns 3157 ns 219021 bytes_per_second=19.3277Gi/s items_per_second=20.753G/s
BatchToTensorSimple<Int8Type>/65536/300 13722 ns 13719 ns 50473 bytes_per_second=4.43975Gi/s items_per_second=4.76715G/s
BatchToTensorSimple<Int8Type>/4194304/3 277670 ns 277521 ns 2510 bytes_per_second=14.0755Gi/s items_per_second=15.1135G/s
BatchToTensorSimple<Int8Type>/4194304/30 293289 ns 293183 ns 2430 bytes_per_second=13.3236Gi/s items_per_second=14.3061G/s
BatchToTensorSimple<Int8Type>/4194304/300 298143 ns 297779 ns 2263 bytes_per_second=13.1179Gi/s items_per_second=14.0853G/s
BatchToTensorSimple<Int16Type>/65536/3 2181 ns 2179 ns 394604 bytes_per_second=28.0054Gi/s items_per_second=15.0353G/s
BatchToTensorSimple<Int16Type>/65536/30 3247 ns 3236 ns 220372 bytes_per_second=18.8588Gi/s items_per_second=10.1247G/s
BatchToTensorSimple<Int16Type>/65536/300 14148 ns 14137 ns 46621 bytes_per_second=4.30854Gi/s items_per_second=2.31313G/s
BatchToTensorSimple<Int16Type>/4194304/3 277347 ns 277092 ns 2553 bytes_per_second=14.0973Gi/s items_per_second=7.56842G/s
BatchToTensorSimple<Int16Type>/4194304/30 370514 ns 323043 ns 2535 bytes_per_second=12.092Gi/s items_per_second=6.49187G/s
BatchToTensorSimple<Int16Type>/4194304/300 297281 ns 296810 ns 2113 bytes_per_second=13.1598Gi/s items_per_second=7.06513G/s
BatchToTensorSimple<Int32Type>/65536/3 2349 ns 2346 ns 387584 bytes_per_second=26.0117Gi/s items_per_second=6.98246G/s
BatchToTensorSimple<Int32Type>/65536/30 3163 ns 3158 ns 213616 bytes_per_second=19.3208Gi/s items_per_second=5.18638G/s
BatchToTensorSimple<Int32Type>/65536/300 13852 ns 13840 ns 49582 bytes_per_second=4.3606Gi/s items_per_second=1.17054G/s
BatchToTensorSimple<Int32Type>/4194304/3 342283 ns 319630 ns 1969 bytes_per_second=12.2212Gi/s items_per_second=3.28059G/s
BatchToTensorSimple<Int32Type>/4194304/30 290756 ns 286728 ns 2381 bytes_per_second=13.6233Gi/s items_per_second=3.65699G/s
BatchToTensorSimple<Int32Type>/4194304/300 300295 ns 297110 ns 2360 bytes_per_second=13.1465Gi/s items_per_second=3.529G/s
BatchToTensorSimple<Int64Type>/65536/3 2204 ns 2197 ns 410967 bytes_per_second=27.7705Gi/s items_per_second=3.7273G/s
BatchToTensorSimple<Int64Type>/65536/30 3176 ns 3162 ns 216236 bytes_per_second=19.3002Gi/s items_per_second=2.59043G/s
BatchToTensorSimple<Int64Type>/65536/300 13656 ns 13588 ns 51372 bytes_per_second=4.4415Gi/s items_per_second=596.128M/s
BatchToTensorSimple<Int64Type>/4194304/3 270131 ns 268433 ns 2622 bytes_per_second=14.552Gi/s items_per_second=1.95313G/s
BatchToTensorSimple<Int64Type>/4194304/30 297324 ns 292629 ns 2026 bytes_per_second=13.3486Gi/s items_per_second=1.79162G/s
BatchToTensorSimple<Int64Type>/4194304/300 293260 ns 291513 ns 2290 bytes_per_second=13.3951Gi/s items_per_second=1.79786G/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1538050070
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t num_cols = state.range(1);
+ const int64_t num_rows = state.range(0) / num_cols / sizeof(CType);
+ arrow::random::RandomArrayGenerator gen_{42};
+
+ std::vector<std::shared_ptr<Field>> fields = {};
+ std::vector<std::shared_ptr<Array>> columns = {};
+
+ for (int64_t i = 0; i < num_cols; ++i) {
+ fields.push_back(field("f" + std::to_string(i), ty));
+ columns.push_back(gen_.ArrayOf(ty, num_rows));
+ }
+ auto schema = std::make_shared<Schema>(std::move(fields));
+ auto batch = RecordBatch::Make(schema, num_rows, columns);
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto tensor, batch->ToTensor());
+ }
+ state.SetItemsProcessed(state.iterations() * num_rows * num_cols);
+ state.SetBytesProcessed(state.iterations() * ty->byte_width() * num_rows * num_cols);
+}
+
+void SetArgs(benchmark::internal::Benchmark* bench) {
+ for (int64_t size : {kL1Size, kL2Size}) {
+ for (int64_t number_of_columns : {3, 30, 300}) {
+ bench->Args({size, number_of_columns});
Review Comment:
Nice =)
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-cow92p3n/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 23.41, 23.22, 14.54
----------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
----------------------------------------------------------------------------------------------------------------------
BatchToTensorSimple<Int8Type>/size:65536/num_columns:3 2486 ns 2485 ns 302338 bytes_per_second=24.5587Gi/s items_per_second=26.3697G/s
BatchToTensorSimple<Int8Type>/size:65536/num_columns:30 3219 ns 3218 ns 217660 bytes_per_second=18.9628Gi/s items_per_second=20.3612G/s
BatchToTensorSimple<Int8Type>/size:65536/num_columns:300 13814 ns 13810 ns 49202 bytes_per_second=4.4103Gi/s items_per_second=4.73553G/s
BatchToTensorSimple<Int8Type>/size:4194304/num_columns:3 297191 ns 297041 ns 2401 bytes_per_second=13.1505Gi/s items_per_second=14.1203G/s
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1538045273
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t num_cols = state.range(1);
+ const int64_t num_rows = state.range(0) / num_cols / sizeof(CType);
+ arrow::random::RandomArrayGenerator gen_{42};
+
+ std::vector<std::shared_ptr<Field>> fields = {};
+ std::vector<std::shared_ptr<Array>> columns = {};
+
+ for (int64_t i = 0; i < num_cols; ++i) {
+ fields.push_back(field("f" + std::to_string(i), ty));
+ columns.push_back(gen_.ArrayOf(ty, num_rows));
+ }
+ auto schema = std::make_shared<Schema>(std::move(fields));
+ auto batch = RecordBatch::Make(schema, num_rows, columns);
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto tensor, batch->ToTensor());
+ }
+ state.SetItemsProcessed(state.iterations() * num_rows * num_cols);
+ state.SetBytesProcessed(state.iterations() * ty->byte_width() * num_rows * num_cols);
+}
+
+void SetArgs(benchmark::internal::Benchmark* bench) {
+ for (int64_t size : {kL1Size, kL2Size}) {
+ for (int64_t number_of_columns : {3, 30, 300}) {
+ bench->Args({size, number_of_columns});
Review Comment:
OK I see, I need to keep them both =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1516157215
##########
cpp/src/arrow/to_tensor_benchmark.cc:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+ auto f0 = field("f0", ty);
+ auto f1 = field("f1", ty);
+ auto f2 = field("f2", ty);
+
+ std::vector<std::shared_ptr<Field>> fields = {f0, f1, f2};
+ auto schema = ::arrow::schema(fields);
+
+ constexpr int kNumRows = 500;
Review Comment:
I have hardcoded the number of rows to `100000` and number of columns to `100`. But I will play around with benchmark templates and `->Apply(Args)` to see if I can figure out how to use them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-1980553789
The result from running `archery benchmark diff --benchmark-filter=BatchToTensorSimple` on the second commit (but with arrays of length `100`, not `500`):
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-jun4cokj/WORKSPACE/build/release/arrow-to-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 24.95, 25.04, 19.14
---------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
---------------------------------------------------------------------------------------------
BatchToTensorSimple<UInt8Type> 550 ns 550 ns 1254345 bytes_per_second=4.06699Gi/s items_per_second=545.863M/s
BatchToTensorSimple<UInt16Type> 555 ns 553 ns 1235570 bytes_per_second=8.08251Gi/s items_per_second=542.408M/s
BatchToTensorSimple<UInt32Type> 569 ns 568 ns 1253335 bytes_per_second=15.7341Gi/s items_per_second=527.949M/s
BatchToTensorSimple<UInt64Type> 580 ns 580 ns 1237449 bytes_per_second=30.8253Gi/s items_per_second=517.163M/s
BatchToTensorSimple<Int8Type> 548 ns 548 ns 1249732 bytes_per_second=4.07944Gi/s items_per_second=547.533M/s
BatchToTensorSimple<Int16Type> 623 ns 568 ns 1233654 bytes_per_second=7.87246Gi/s items_per_second=528.312M/s
BatchToTensorSimple<Int32Type> 565 ns 564 ns 1204923 bytes_per_second=15.8461Gi/s items_per_second=531.706M/s
BatchToTensorSimple<Int64Type> 585 ns 585 ns 1269059 bytes_per_second=30.5699Gi/s items_per_second=512.878M/s
BatchToTensorSimple<HalfFloatType> 545 ns 544 ns 1217900 bytes_per_second=8.21219Gi/s items_per_second=551.111M/s
BatchToTensorSimple<FloatType> 575 ns 574 ns 1239991 bytes_per_second=15.5835Gi/s items_per_second=522.896M/s
BatchToTensorSimple<DoubleType> 567 ns 566 ns 1152074 bytes_per_second=31.5943Gi/s items_per_second=530.065M/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1535512580
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = state.range(0) / sizeof(CType);
Review Comment:
```suggestion
const int64_t num_cols = state.range(1);
const int64_t num_rows = state.range(0) / num_cols / sizeof(CType);
```
To get the number of rows, I would divide the total size by the number of columns as well, so the total size (and timing) for each benchmark is more or less the same (right now the case with 3 columns is really small time, while the one with 300 columns probably takes a bit long).
And I think we can also use normal variable names here, the `k` prefix is for global constants, but here I would say it's just a local variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40358:
URL: https://github.com/apache/arrow/pull/40358#issuecomment-2014950925
@pitrou I have included your suggestions.
This is the output with the latest changes:
```
Running /var/folders/gw/q7wqd4tx18n_9t4kbkd0bj1m0000gn/T/arrow-archery-s2l7kna2/WORKSPACE/build/release/arrow-tensor-benchmark
Run on (8 X 24 MHz CPU s)
CPU Caches:
L1 Data 64 KiB
L1 Instruction 128 KiB
L2 Unified 4096 KiB (x8)
Load Average: 19.25, 15.77, 10.23
-----------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------------
BatchToTensorSimple<Int8Type>/65536/3 3803 ns 3802 ns 179651 bytes_per_second=48.1626Gi/s items_per_second=51.7142G/s
BatchToTensorSimple<Int8Type>/65536/30 141770 ns 140390 ns 5332 bytes_per_second=13.0426Gi/s items_per_second=14.0044G/s
BatchToTensorSimple<Int8Type>/65536/300 1509271 ns 1488588 ns 471 bytes_per_second=12.3006Gi/s items_per_second=13.2077G/s
BatchToTensorSimple<Int8Type>/4194304/3 892806 ns 890394 ns 792 bytes_per_second=13.1613Gi/s items_per_second=14.1318G/s
BatchToTensorSimple<Int8Type>/4194304/30 10833571 ns 10319294 ns 68 bytes_per_second=11.3562Gi/s items_per_second=12.1936G/s
BatchToTensorSimple<Int8Type>/4194304/300 1128155000 ns 551951000 ns 1 bytes_per_second=2.12315Gi/s items_per_second=2.27972G/s
BatchToTensorSimple<Int16Type>/65536/3 3781 ns 3769 ns 185929 bytes_per_second=48.5878Gi/s items_per_second=26.0854G/s
BatchToTensorSimple<Int16Type>/65536/30 129794 ns 129615 ns 5636 bytes_per_second=14.1269Gi/s items_per_second=7.58431G/s
BatchToTensorSimple<Int16Type>/65536/300 1553976 ns 1550687 ns 435 bytes_per_second=11.808Gi/s items_per_second=6.33938G/s
BatchToTensorSimple<Int16Type>/4194304/3 824934 ns 822791 ns 882 bytes_per_second=14.2427Gi/s items_per_second=7.64648G/s
BatchToTensorSimple<Int16Type>/4194304/30 9991414 ns 9954623 ns 69 bytes_per_second=11.7722Gi/s items_per_second=6.32013G/s
BatchToTensorSimple<Int16Type>/4194304/300 791524063 ns 310795500 ns 2 bytes_per_second=3.77057Gi/s items_per_second=2.02431G/s
BatchToTensorSimple<Int32Type>/65536/3 3717 ns 3712 ns 183626 bytes_per_second=49.3228Gi/s items_per_second=13.24G/s
BatchToTensorSimple<Int32Type>/65536/30 135493 ns 134325 ns 5035 bytes_per_second=13.6315Gi/s items_per_second=3.65918G/s
BatchToTensorSimple<Int32Type>/65536/300 1607824 ns 1600713 ns 436 bytes_per_second=11.439Gi/s items_per_second=3.07063G/s
BatchToTensorSimple<Int32Type>/4194304/3 863068 ns 860123 ns 782 bytes_per_second=13.6245Gi/s items_per_second=3.6573G/s
BatchToTensorSimple<Int32Type>/4194304/30 10307080 ns 10272412 ns 68 bytes_per_second=11.408Gi/s items_per_second=3.06231G/s
BatchToTensorSimple<Int32Type>/4194304/300 261872267 ns 147986600 ns 5 bytes_per_second=7.91879Gi/s items_per_second=2.12568G/s
BatchToTensorSimple<Int64Type>/65536/3 3725 ns 3722 ns 183079 bytes_per_second=49.1992Gi/s items_per_second=6.60341G/s
BatchToTensorSimple<Int64Type>/65536/30 126616 ns 126444 ns 5720 bytes_per_second=14.4811Gi/s items_per_second=1.94362G/s
BatchToTensorSimple<Int64Type>/65536/300 1508292 ns 1506162 ns 445 bytes_per_second=12.1571Gi/s items_per_second=1.6317G/s
BatchToTensorSimple<Int64Type>/4194304/3 837330 ns 835840 ns 833 bytes_per_second=14.0203Gi/s items_per_second=1.88178G/s
BatchToTensorSimple<Int64Type>/4194304/30 9866716 ns 9823261 ns 69 bytes_per_second=11.9296Gi/s items_per_second=1.60116G/s
BatchToTensorSimple<Int64Type>/4194304/300 745086639 ns 325750333 ns 3 bytes_per_second=3.59746Gi/s items_per_second=482.843M/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40357: [C++] Add benchmark for ToTensor conversions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40358:
URL: https://github.com/apache/arrow/pull/40358#discussion_r1533911075
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
Review Comment:
```suggestion
```
##########
cpp/src/arrow/tensor_benchmark.cc:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "benchmark/benchmark.h"
+
+#include "arrow/record_batch.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+#include "arrow/type.h"
+#include "arrow/util/benchmark_util.h"
+
+namespace arrow {
+
+template <typename ValueType>
+static void BatchToTensorSimple(benchmark::State& state) {
+ RegressionArgs args(state);
+ using CType = typename ValueType::c_type;
+ std::shared_ptr<DataType> ty = TypeTraits<ValueType>::type_singleton();
+
+ const int64_t kNumRows = args.size / sizeof(CType);
Review Comment:
```suggestion
const int64_t kNumRows = state.range(0) / sizeof(CType);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org