You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2025/10/14 13:12:46 UTC
[I] [C++] Continuous benchmarking failures [arrow]
pitrou opened a new issue, #47817:
URL: https://github.com/apache/arrow/issues/47817
### Describe the bug, including details regarding any error messages, version, and platform.
From https://buildkite.com/apache-arrow it seems that 2 of the 4 continuous benchmarking platforms have been failing for quite some time, and they fail on the C++ benchmarks.
### Component(s)
C++, Benchmarking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3403158060
Checked the https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux benchmark:
* Last successful run at 3rd Oct 2025 at 21:48 UTC: https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/builds/1591/steps/canvas
* First failed run at 6th Oct 2025 at 01:18 UTC: https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/builds/1592/steps/canvas
There is only one commit 4b55ea69574a7b35f545d27e7423e8a88a7e256a between this period. Maybe it is worth a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #47817: [C++] Continuous benchmarking failures
URL: https://github.com/apache/arrow/issues/47817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3402605719
I've tried to look for clues in the [logs](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/builds/1634/steps/canvas?sid=0199e357-8fa8-469b-b7cc-3b6b6f026ade) but couldn't find any obvious issue.
The one thing that I noticed is that the benchmark executable that fails starting is also the last one being built by CMake:
```
[662/662] : && /var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/x86_64-conda-linux-gnu-c++ -Wredundant-move -Wno-noexcept-type -Wno-self-move -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /var/lib/buildkite-agent/.conda/envs/arrow-commit/include -fdiagnostics-color=always -Wall -Wno-conversion -Wno-sign-conversion -Wdate-time -Wimplicit-fallthrough -Wunused-result -fno-semantic-interposition -msse4.2 -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib -Wl,-rpath-link,/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib -L/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib -Wl,--dependency-file=src/arrow/dataset/CMakeFiles/arrow-dataset-scanner-benchmark.dir/link.d src
/arrow/dataset/CMakeFiles/arrow-dataset-scanner-benchmark.dir/__/compute/benchmark_main.cc.o src/arrow/dataset/CMakeFiles/arrow-dataset-scanner-benchmark.dir/scanner_benchmark.cc.o -o release/arrow-dataset-scanner-benchmark -Wl,-rpath,/tmp/arrow-cpp-ae5b1b8c-235f-4036-81e5-3a2d54c9fbf7/cpp/release /var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/libbenchmark_main.so.1.9.0 release/libarrow_testing.so.2200.0.0 release/libarrow_gmock_main.so.1.16.0 release/libarrow_dataset.so.2200.0.0 /var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/libbenchmark.so.1.9.0 release/libarrow_gmock.so.1.16.0 release/libarrow_gtest.so.1.16.0 release/libparquet.so.2200.0.0 release/libarrow_acero.so.2200.0.0 release/libarrow_compute.so.2200.0.0 release/libarrow.so.2200.0.0 && :
```
But I don't know why that would be a problem (unless there are disk space issues?). cc @assignUser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3401753665
@zanmato1984 @kou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3402968274
No I don't think that be an issue, Any failures should still surface, same for disk space issues...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3403375436
buildkite should pull the conda envs from the arrow repo so that change should be reflected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3401752823
The only thing I can gather from the benchmarking logs is that `arrow-dataset-scanner-benchmark --benchmark_list_tests` seems to fail with status code (apparently this can mean that the executable was not found):
```
251014-12:58:23.266 INFO: Initializing adapter
251014-12:58:23.302 INFO: Initializing adapter
251014-12:58:23.302 INFO: Running benchmarks with command: `archery benchmark run --output /tmp/tmpakevgp2o --repetitions 6 --repetition-min-time 0.05 --suite-filter ^(?!arrow-acero-aggregate-benchmark)(?!arrow-filesystem-s3fs-benchmark).*$ /tmp/arrow-cpp-d3f65fe9-63d8-43e0-a229-79f30295eeba/cpp`
Traceback (most recent call last):
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/archery", line 7, in <module>
sys.exit(archery())
^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1462, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1383, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1850, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1850, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1246, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 814, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/cli.py", line 467, in benchmark_run
json_out = json.dumps(runner_base, cls=JsonEncoder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/json/__init__.py", line 238, in dumps
**kw).encode(obj)
^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/codec.py", line 35, in default
return BenchmarkRunnerCodec.encode(o)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/codec.py", line 79, in encode
return {"suites": [BenchmarkSuiteCodec.encode(s) for s in br.suites]}
^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/runner.py", line 183, in suites
suite = self.suite(suite_name, suite_bin)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/runner.py", line 153, in suite
benchmark_names = suite_cmd.list_benchmarks()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/benchmark/google.py", line 49, in list_benchmarks
result = self.run(*argv, stdout=subprocess.PIPE,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/dev/archery/archery/utils/command.py", line 78, in run
return subprocess.run(invocation, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/arrow-cpp-d3f65fe9-63d8-43e0-a229-79f30295eeba/cpp/release/arrow-dataset-scanner-benchmark', '--benchmark_list_tests']' returned non-zero exit status 127.
Traceback (most recent call last):
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/conbench", line 7, in <module>
sys.exit(conbench())
^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1462, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1383, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1850, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 1246, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/click/core.py", line 814, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/conbenchlegacy/cli.py", line 149, in _benchmark
for result, output in benchmark().run(**kwargs):
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-01490802ac0311209-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/benchmarks/benchmarks/cpp_micro_benchmarks.py", line 136, in run
self.adapter.run(command_params)
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/site-packages/benchadapt/adapters/_adapter.py", line 98, in run
subprocess.run(args=command, check=True)
File "/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['archery', 'benchmark', 'run', '--output', PosixPath('/tmp/tmpakevgp2o'), '--repetitions', '6', '--repetition-min-time', '0.05', '--suite-filter', '^(?!arrow-acero-aggregate-benchmark)(?!arrow-filesystem-s3fs-benchmark).*$', '/tmp/arrow-cpp-d3f65fe9-63d8-43e0-a229-79f30295eeba/cpp']' returned non-zero exit status 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Continuous benchmarking failures [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #47817:
URL: https://github.com/apache/arrow/issues/47817#issuecomment-3403370373
Yeah, the benchmarks use conda so that might be the cause.
> 127 (apparently this can mean that the executable was not found)
I think that's a red hering (iirc docker uses 127 too for failed command execution)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org