You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "austin3dickey (via GitHub)" <gi...@apache.org> on 2024/04/01 20:14:34 UTC

[PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

austin3dickey opened a new pull request, #40925:
URL: https://github.com/apache/arrow/pull/40925

   Previously, we were relying on `archery` to build C++ microbenchmarks during Conbench runs. That doubled the build time and ignored all the environment variables set in this file. Along with https://github.com/voltrondata-labs/benchmarks/pull/159, this change will remove the duplicated build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2037534389

   @ursabot please benchmark lang=C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2030892909

   Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 2a0788fe3eb2574c923f450dd2c3ed9e410d00fe.
   
   None of the specified runs were found on the Conbench server.
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23324031848) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2037534578

   Benchmark runs are scheduled for commit b7440f12ad5d58f896098898263c9e4e4305a50c. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2047674244

   Requesting review from @pitrou as well.
   
   It looks like this change does cause slight regressions in some C++ benchmarks. Here's [an example](https://conbench.ursa.dev/benchmark-results/06615ace7d9d7cb0800055526a223269/). I looked through the graphs, and it looks like in most cases the regressions simply offset the improvement we got from upgrading to the GNU 12.3.0 compiler back on July 11. I think the regressions make sense because we're now building Arrow differently, with different cmake options. This should be fine and is a reflection of the environment setup, not the code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2047877915

   >I think you can disable...
   
   Cool, I'll try that out. Note that with this change we started running these additional suites. Not sure exactly what flags turned them on, but it's probably nice to run them if we can.
   - arrow-range-benchmark
   - arrow-compression-benchmark
   - arrow-machine-benchmark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on code in PR #40925:
URL: https://github.com/apache/arrow/pull/40925#discussion_r1558238148


##########
dev/archery/archery/benchmark/google.py:
##########
@@ -57,7 +57,7 @@ def results(self, repetitions=1, repetition_min_time=None):
                     "--benchmark_out_format=json"]
 
             if repetition_min_time is not None:
-                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}")
+                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}s")

Review Comment:
   This was to suppress a message I saw in the logs:
   ```
   LOG(0): Value passed to --benchmark_min_time should have a suffix. Eg., `30s` for 30-seconds.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2037556908

   Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit b7440f12ad5d58f896098898263c9e4e4305a50c.
   
   None of the specified runs were found on the Conbench server.
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23449774987) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #40925:
URL: https://github.com/apache/arrow/pull/40925#discussion_r1559560291


##########
dev/archery/archery/benchmark/google.py:
##########
@@ -57,7 +57,7 @@ def results(self, repetitions=1, repetition_min_time=None):
                     "--benchmark_out_format=json"]
 
             if repetition_min_time is not None:
-                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}")
+                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}s")

Review Comment:
   Yes, it's a change in the Google benchmark CLI.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2045962651

   Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 6961a0e2987190dbfafddb213fed5b35567b37e0.
   
   None of the specified runs were found on the Conbench server.
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23630784576) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2030495644

   Benchmark runs are scheduled for commit 2a0788fe3eb2574c923f450dd2c3ed9e410d00fe. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41133: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2048424969

   Though it is not a MINOR, I have created an issue. @austin3dickey please comment 'Take' on it to get it assigned to you (gh anti-spam prevents me from assigning you if you haven't interacted with it)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41133: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser merged PR #40925:
URL: https://github.com/apache/arrow/pull/40925


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #40925:
URL: https://github.com/apache/arrow/pull/40925#discussion_r1560043595


##########
dev/archery/archery/benchmark/google.py:
##########
@@ -57,7 +57,7 @@ def results(self, repetitions=1, repetition_min_time=None):
                     "--benchmark_out_format=json"]
 
             if repetition_min_time is not None:
-                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}")
+                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}s")

Review Comment:
   It seems that the warning was added with the flag itself https://github.com/google/benchmark/blame/70916cbf71f50b9e1e6f13559e10d6dbb92beb32/src/benchmark_runner.cc#L199 so this isn't an issue after all



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2033000625

   Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 4a2d2123518ebcdf9ed19de708283bfa5be86088.
   
   None of the specified runs were found on the Conbench server.
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23363013567) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2032983323

   @ursabot please benchmark lang=C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2045958928

   @ursabot please benchmark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2048166656

   @ursabot please benchmark lang=C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2034640604

   @ursabot please benchmark lang=C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2045948136

   Benchmark runs are scheduled for commit 6961a0e2987190dbfafddb213fed5b35567b37e0. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on code in PR #40925:
URL: https://github.com/apache/arrow/pull/40925#discussion_r1558238148


##########
dev/archery/archery/benchmark/google.py:
##########
@@ -57,7 +57,7 @@ def results(self, repetitions=1, repetition_min_time=None):
                     "--benchmark_out_format=json"]
 
             if repetition_min_time is not None:
-                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}")
+                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}s")

Review Comment:
   This was to suppress a message I saw in the logs:
   ```
   LOG(0): Value passed to --benchmark_min_time should have a suffix. Eg., `30s` for 30-seconds.
   ```
   
   It's already documented that this value should be seconds:
   
   https://github.com/apache/arrow/blob/831b94a65ebaaabe6fcad0ea378d0a784ef5bbb7/dev/archery/archery/cli.py#L450



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41133: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2048712420

   After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 3105821498030f860b8e0a7292b28a893b4a23c3.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23687062483) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2034641926

   Benchmark runs are scheduled for commit 88bda24bc1891731f20418331251aedcd452a8fb. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2047741181

   > I think the regressions make sense because we're now building Arrow differently, with different cmake options. This should be fine and is a reflection of the environment setup, not the code.
   
   This sounds fine to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2046224064

   Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit 615e38fe479e14a7f270a6a1eff7de2e9bf94cc0.
   
   There were 36 benchmark results indicating a performance regression:
   
   - Pull Request Run on `arm64-t4g-linux-compute` at [2024-04-09 20:58:37Z](https://conbench.ursa.dev/compare/runs/af380eb4a082407cbe572847bf1933a4...aa1ae8777ec0460f98a6e7cd119b5751/)
     - [`ShortVectorInsertAtEnd` (C++) with params=<STATIC_VECTOR(int)>, source=cpp-micro, suite=arrow-small-vector-benchmark](https://conbench.ursa.dev/compare/benchmarks/0660aeb9a5d97ffd8000641bf9b18d82...06615af58349704c8000cf2f83d5c538)
     - [`CopyShortVector` (C++) with params=<SMALL_VECTOR(std::shared_ptr<int>)>, source=cpp-micro, suite=arrow-small-vector-benchmark](https://conbench.ursa.dev/compare/benchmarks/0660aeb95c5e78db800054d92f245144...06615af5656f73ad800055533f27545c)
   - and 34 more (see the report linked below)
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23637882154) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on code in PR #40925:
URL: https://github.com/apache/arrow/pull/40925#discussion_r1559515090


##########
dev/conbench_envs/benchmarks.env:
##########
@@ -27,6 +28,7 @@ ARROW_GANDIVA=OFF
 ARROW_HDFS=ON
 ARROW_HOME=$CONDA_PREFIX
 ARROW_INSTALL_NAME_RPATH=ON
+ARROW_JEMALLOC=OFF

Review Comment:
   This used to be set [here](https://github.com/voltrondata-labs/arrow-benchmarks-ci/commit/7c20a64e464705d5bf4e2f20f4c8befd601d566a) but I think it belongs in this file instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2045959114

   Benchmark runs are scheduled for commit 615e38fe479e14a7f270a6a1eff7de2e9bf94cc0. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2032983566

   Benchmark runs are scheduled for commit 4a2d2123518ebcdf9ed19de708283bfa5be86088. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2047758359

   I think you can disable `ARROW_BUILD_BENCHMARKS_REFERENCE` otherwise you'll get more unstable benchmark results. You could probably also disable the examples, executables, utilities, as well as S3 and ORC.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2048282185

   Thanks for your patience. Conbench analyzed the 2 benchmarking runs that have been run so far on PR commit 648723d74625cfef7f4b7e13787da4de3b164794.
   
   There were 50 benchmark results indicating a performance regression:
   
   - Pull Request Run on `arm64-m6g-linux-compute` at [2024-04-10 18:55:51Z](https://conbench.ursa.dev/compare/runs/10c555696d734eb1aabaf6051c0eb094...f73aa1a4e9e44d9f848a7d400b43681f/)
     - [`ShortVectorInsertAtEnd` (C++) with params=<SMALL_VECTOR(int)>, source=cpp-micro, suite=arrow-small-vector-benchmark](https://conbench.ursa.dev/compare/benchmarks/06615a8c4a7f78f38000738b09bbb99c...06616e411ed071e680002a65c8265bb5)
     - [`ThreadPoolSpawn` (C++) with params=threads:2/task_cost:100000/real_time, source=cpp-micro, suite=arrow-thread-pool-benchmark](https://conbench.ursa.dev/compare/benchmarks/06615a8c57c87a038000b5d99a4e80a5...06616e41341e706f8000b2003240148e)
   - and 48 more (see the report linked below)
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23677179232) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41133: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2048424721

   :warning: GitHub issue #41133 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2034653304

   Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 88bda24bc1891731f20418331251aedcd452a8fb.
   
   None of the specified runs were found on the Conbench server.
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23395234287) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2030494789

   @ursabot please benchmark lang=C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #40925:
URL: https://github.com/apache/arrow/pull/40925#discussion_r1559562002


##########
dev/archery/archery/benchmark/google.py:
##########
@@ -57,7 +57,7 @@ def results(self, repetitions=1, repetition_min_time=None):
                     "--benchmark_out_format=json"]
 
             if repetition_min_time is not None:
-                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}")
+                argv.append(f"--benchmark_min_time={repetition_min_time:.6f}s")

Review Comment:
   Note that this might not work on old versions of Google benchmark, so it's worth being careful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2047742121

   For reference, here's the full diff of cmake flags:
   
   <details>
   
   ```
   > diff before.txt after.txt
   10c10
   <    ARROW_GIT_ID=835e21873522bba2928beb972e863790f4c270cb [default=""]
   ---
   >    ARROW_GIT_ID=615e38fe479e14a7f270a6a1eff7de2e9bf94cc0 [default=""]
   12c12
   <    ARROW_GIT_DESCRIPTION=apache-arrow-16.0.0.dev-483-g835e21873 [default=""]
   ---
   >    ARROW_GIT_DESCRIPTION=apache-arrow-16.0.0.dev-422-g615e38fe4 [default=""]
   14c14
   <    ARROW_NO_DEPRECATED_API=OFF [default=OFF]
   ---
   >    ARROW_NO_DEPRECATED_API=ON [default=OFF]
   23c23
   <    ARROW_USE_LD_GOLD=ON [default=OFF]
   ---
   >    ARROW_USE_LD_GOLD=OFF [default=OFF]
   25,26d24
   <    ARROW_USE_LLD=OFF [default=OFF]
   <        Use the LLVM lld for linking (if available)
   60c58
   <    ARROW_BUILD_BENCHMARKS_REFERENCE=OFF [default=OFF]
   ---
   >    ARROW_BUILD_BENCHMARKS_REFERENCE=ON [default=OFF]
   99c97
   <    ARROW_BUILD_UTILITIES=OFF [default=OFF]
   ---
   >    ARROW_BUILD_UTILITIES=ON [default=OFF]
   119c117
   <    ARROW_HDFS=OFF [default=OFF]
   ---
   >    ARROW_HDFS=ON [default=OFF]
   127c125
   <    ARROW_MIMALLOC=OFF [default=OFF]
   ---
   >    ARROW_MIMALLOC=ON [default=OFF]
   131c129
   <    ARROW_ORC=OFF [default=OFF]
   ---
   >    ARROW_ORC=ON [default=OFF]
   136c134
   <    ARROW_S3=OFF [default=OFF]
   ---
   >    ARROW_S3=ON [default=OFF]
   149c147
   <    ARROW_DEPENDENCY_SOURCE=CONDA [default=CONDA|AUTO|BUNDLED|SYSTEM|VCPKG|BREW]
   ---
   >    ARROW_DEPENDENCY_SOURCE=AUTO [default=CONDA|AUTO|BUNDLED|SYSTEM|VCPKG|BREW]
   216c214
   <    PARQUET_BUILD_EXECUTABLES=OFF [default=OFF]
   ---
   >    PARQUET_BUILD_EXECUTABLES=ON [default=OFF]
   218c216
   <    PARQUET_BUILD_EXAMPLES=OFF [default=OFF]
   ---
   >    PARQUET_BUILD_EXAMPLES=ON [default=OFF]
   220c218
   <    PARQUET_REQUIRE_ENCRYPTION=OFF [default=OFF]
   ---
   >    PARQUET_REQUIRE_ENCRYPTION=ON [default=OFF]
   228c226
   <    ARROW_GANDIVA_PC_CXX_FLAGS="" [default=""]
   ---
   >    ARROW_GANDIVA_PC_CXX_FLAGS=-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../lib/gcc/aarch64-conda-linux-gnu/12.3.0/include;-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../lib/gcc/aarch64-conda-linux-gnu/12.3.0/include-fixed;-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../lib/gcc/aarch64-conda-linux-gnu/12.3.0/../../../../aarch64-conda-linux-gnu/include;-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../lib/gcc/../../aarch64-conda-linux-gnu/include/c++/12.3.0;-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../lib/gcc/../../aarch64-conda-linux-gnu/include/c++/12.3.0/aarch64-conda-linux-gnu;-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../lib/gcc/../../aarch64-conda-linux-gnu/include/c++/12.3.0/backward;-isystem;/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/../aarch64-conda-linux-gnu/sysroot/usr/include; [default=""]
   ```
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2048166747

   Benchmark runs are scheduled for commit 648723d74625cfef7f4b7e13787da4de3b164794. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: [Benchmarking] Build benchmarks in benchmarks.env [arrow]

Posted by "austin3dickey (via GitHub)" <gi...@apache.org>.
austin3dickey commented on PR #40925:
URL: https://github.com/apache/arrow/pull/40925#issuecomment-2045948052

   @ursabot please benchmark lang=C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org