You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/09/17 15:57:34 UTC
[GitHub] [arrow] kiszk opened a new pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
kiszk opened a new pull request #8210:
URL: https://github.com/apache/arrow/pull/8210
This PR supports Java benchmark in Ursabot. The implementation is based on [this suggestion](https://mail-archives.apache.org/mod_mbox/arrow-dev/202008.mbox/%3cCABNn7+q35j7QWsHJBX8omdewKT+F1p_M7r1_F6szs4dqc+Luyg@mail.gmail.com%3e)
Here are work items.
- [x ] Support `--language=[cpp|java]` option
- [ ] Enable to build java binding
- [ ] Enable to run Java benchmarks
- [ ] Allows us to filter/select benchmarks
- [ ] Enable to collect results
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-817856268
Here is an example.
```
% archery benchmark diff --language=java --benchmark-filter="setWith" HEAD HEAD~1
...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (2)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
benchmark baseline contender change % configurations
org.apache.arrow.vector.IntBenchmarks.setWithValueHolder 165.925K items/sec 165.898K items/sec -0.016 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 1, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
org.apache.arrow.vector.IntBenchmarks.setWithWriter 222.780K items/sec 221.768K items/sec -0.454 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 1, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-726544148
The following commands should work:
```
archery benchmark list --langauge=java
archery benchmark run --langauge=java
archery benchmark diff --langauge=java
```
Here is an example
```
archery benchmark run ARROW-10031 --output=out-java.json --language=java --benchmark-filter=IntBench --java-options=-mx1g --build-extras=-Dfoo.bar=1 --benchmark-extras=-Dbar.foo=2
````
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on a change in pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
bkietz commented on a change in pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#discussion_r555119488
##########
File path: dev/archery/archery/benchmark/compare.py
##########
@@ -54,6 +54,14 @@ def formatter_for_unit(unit):
return bytes_per_seconds_fmt
elif unit == "items_per_second":
return items_per_seconds_fmt
+ elif unit.startswith("ops/"):
Review comment:
Instead, please convert to "items_per_second" or "bytes_per_second" in `JavaMicrobenchmarkHarnessObservation.unit`
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
+ """ Run a Java Micro Benchmark Harness
+
+ This assumes the binary supports the standard command line options,
+ notably `--benchmark_filter`, `--benchmark_format`, etc...
+ """
+
+ def __init__(self, build, benchmark_filter=None):
+ self.benchmark_filter = benchmark_filter
+ self.build = build
+ self.maven = Maven()
+
+ def list_benchmarks(self):
+ argv = []
+ if self.benchmark_filter:
+ argv.append("-Dbenchmark.filter={}".format(self.benchmark_filter))
+ result = self.build.list(
+ *argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ """ Extract benchmark names from output. Assume the following output
Review comment:
Please put docstrings on the first line of function definitions
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
+ """ Run a Java Micro Benchmark Harness
+
+ This assumes the binary supports the standard command line options,
+ notably `--benchmark_filter`, `--benchmark_format`, etc...
+ """
+
+ def __init__(self, build, benchmark_filter=None):
+ self.benchmark_filter = benchmark_filter
+ self.build = build
+ self.maven = Maven()
+
+ def list_benchmarks(self):
+ argv = []
+ if self.benchmark_filter:
+ argv.append("-Dbenchmark.filter={}".format(self.benchmark_filter))
+ result = self.build.list(
+ *argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ """ Extract benchmark names from output. Assume the following output
+ ...
+ Benchmarks:
+ org.apache.arrow.vector.IntBenchmarks.setIntDirectly
+ ...
+ org.apache.arrow.vector.IntBenchmarks.setWithValueHolder
+ org.apache.arrow.vector.IntBenchmarks.setWithWriter
+ ...
+ [INFO]
+ """
+
+ lists = []
+ benchmarks = False
+ for line in str.splitlines(result.stdout.decode("utf-8")):
+ if not benchmarks:
+ if line.startswith("Benchmarks:"):
+ benchmarks = True
+ else:
+ if line.startswith("org.apache.arrow"):
+ lists.append(line)
+ if line.startswith("[INFO]"):
+ benchmarks = False
Review comment:
```suggestion
```
##########
File path: dev/archery/archery/cli.py
##########
@@ -367,11 +381,21 @@ def benchmark_common_options(cmd):
click.option("--output", metavar="<output>",
type=click.File("w", encoding="utf8"), default="-",
help="Capture output result into file."),
+ click.option("--language", metavar="<lang>", type=str, default="cpp",
+ show_default=True, callback=check_language,
+ help="Specify target language for the benchmark"),
+ click.option("--build-extras", type=str, multiple=True,
+ help="Extra flags/options to pass to mvn build. "
+ "Can be stackd. For language=java"),
Review comment:
```suggestion
"Can be stacked. For language=java"),
```
##########
File path: dev/archery/archery/cli.py
##########
@@ -394,16 +418,53 @@ def benchmark_filter_options(cmd):
@benchmark_common_options
@click.pass_context
def benchmark_list(ctx, rev_or_path, src, preserve, output, cmake_extras,
- **kwargs):
+ language, build_extras, benchmark_extras, java_home,
+ java_options, **kwargs):
""" List benchmark suite.
"""
+ if language == "cpp":
+ ctx.forward(benchmark_list_cpp, **kwargs)
+ elif language == "java":
+ ctx.forward(benchmark_list_java, **kwargs)
+
+
+@benchmark.command(name="list_cpp")
+@click.pass_context
+def benchmark_list_cpp(ctx, rev_or_path, src, preserve, output, cmake_extras,
Review comment:
This division seems unnecessary. Please just keep a single `benchmark_list` function (moving the branch on language into the `with` block)
##########
File path: dev/archery/archery/lang/java.py
##########
@@ -28,3 +31,48 @@ def __init__(self, jar, *args, **kwargs):
self.jar = jar
self.argv = ("-jar", jar)
Java.__init__(self, *args, **kwargs)
+
+
+class JavaConfiguration:
+ def __init__(self,
+
+ # toolchain
+ java_home=None, java_options=None,
+ # build & benchmark
+ build_extras=None, benchmark_extras=None):
+ self.java_home = java_home
+ self.java_options = java_options
+
+ self.build_extras = build_extras
+ self.benchmark_extras = benchmark_extras
+
+ @property
+ def build_definitions(self):
+ extras = list(self.build_extras) if self.build_extras else []
+ return extras
Review comment:
I'd recommend doing the cast-to-list here; if an iterator were passed for `build_extras` then it'd be depleted after the first access to `build_definitions`:
```suggestion
self.build_extras = list(build_extras) if build_extras else []
self.benchmark_extras = list(benchmark_extras) if benchmark_extras else []
@property
def build_definitions(self):
return self.build_extras
```
I see that CppConfiguration has this same flaw; it's not necessary for you to address it there too
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
+ """ Run a Java Micro Benchmark Harness
+
+ This assumes the binary supports the standard command line options,
+ notably `--benchmark_filter`, `--benchmark_format`, etc...
+ """
+
+ def __init__(self, build, benchmark_filter=None):
+ self.benchmark_filter = benchmark_filter
+ self.build = build
+ self.maven = Maven()
+
+ def list_benchmarks(self):
+ argv = []
+ if self.benchmark_filter:
+ argv.append("-Dbenchmark.filter={}".format(self.benchmark_filter))
+ result = self.build.list(
+ *argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ """ Extract benchmark names from output. Assume the following output
Review comment:
Also, please make it clear that you're using the strings "Benchmarks:" and "[INFO]" to delimit lines containing benchmark names
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
Review comment:
It's odd that this class inherits `Maven` instead of `Command`, especially since it has a member `self.maven`
```suggestion
class JavaMicrobenchmarkHarnessCommand(Command):
```
##########
File path: dev/archery/archery/cli.py
##########
@@ -357,6 +366,11 @@ def benchmark(ctx):
def benchmark_common_options(cmd):
+ def check_language(ctx, param, value):
+ if value != "cpp" and value != "java":
Review comment:
```suggestion
if value not in {"cpp", "java"}:
```
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
+ """ Run a Java Micro Benchmark Harness
+
+ This assumes the binary supports the standard command line options,
+ notably `--benchmark_filter`, `--benchmark_format`, etc...
Review comment:
This comment doesn't apply to java benchmarks; seems copy pasted from GoogleBenchmarkCommand?
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
+ """ Run a Java Micro Benchmark Harness
+
+ This assumes the binary supports the standard command line options,
+ notably `--benchmark_filter`, `--benchmark_format`, etc...
+ """
+
+ def __init__(self, build, benchmark_filter=None):
+ self.benchmark_filter = benchmark_filter
+ self.build = build
+ self.maven = Maven()
+
+ def list_benchmarks(self):
+ argv = []
+ if self.benchmark_filter:
+ argv.append("-Dbenchmark.filter={}".format(self.benchmark_filter))
+ result = self.build.list(
+ *argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ """ Extract benchmark names from output. Assume the following output
+ ...
+ Benchmarks:
+ org.apache.arrow.vector.IntBenchmarks.setIntDirectly
+ ...
+ org.apache.arrow.vector.IntBenchmarks.setWithValueHolder
+ org.apache.arrow.vector.IntBenchmarks.setWithWriter
+ ...
+ [INFO]
+ """
+
+ lists = []
+ benchmarks = False
+ for line in str.splitlines(result.stdout.decode("utf-8")):
+ if not benchmarks:
+ if line.startswith("Benchmarks:"):
+ benchmarks = True
+ else:
+ if line.startswith("org.apache.arrow"):
+ lists.append(line)
+ if line.startswith("[INFO]"):
+ benchmarks = False
+ break
+ return lists
+
+ def results(self, repetitions=1):
+ with NamedTemporaryFile(suffix=".json") as out:
+ argv = ["-Dbenchmark.runs={}".format(repetitions),
+ "-Dbenchmark.resultfile={}".format(out.name),
+ "-Dbenchmark.resultformat=json"]
+ if self.benchmark_filter:
+ argv.append(
+ "-Dbenchmark.filter={}".format(self.benchmark_filter)
+ )
+
+ self.build.benchmark(*argv, check=True)
+ return json.load(out)
+
+
+class JavaMicrobenchmarkHarnessObservation:
+ """ Represents one run of a single Java Microbenchmark Harness
+ """
+
+ def __init__(self, benchmark, primaryMetric,
+ forks, warmupIterations, measurementIterations, **counters):
+ self.name = benchmark
+ self.primaryMetric = primaryMetric
+ self.score = primaryMetric["score"]
+ self.scoreUnit = primaryMetric["scoreUnit"]
+ self.forks = forks
+ self.warmups = warmupIterations
+ self.runs = measurementIterations
+ self.counters = {
+ "mode": counters["mode"],
+ "threads": counters["threads"],
+ "warmups": warmupIterations,
+ "warmupTime": counters["warmupTime"],
+ "measurements": measurementIterations,
+ "measurementTime": counters["measurementTime"],
+ "jvmArgs": counters["jvmArgs"]
+ }
+
+ @property
+ def value(self):
+ """ Return the benchmark value."""
+ return self.score
+
+ @property
+ def unit(self):
+ return self.scoreUnit
+
+ def __repr__(self):
+ return str(self.value)
+
+
+class JavaMicrobenchmarkHarness(Benchmark):
+ """ A set of JavaMicrobenchmarkHarnessObservations. """
+
+ def __init__(self, name, runs):
+ """ Initialize a JavaMicrobenchmarkHarness.
+
+ Parameters
+ ----------
+ name: str
+ Name of the benchmark
+ forks: int
+ warmups: int
+ runs: int
+ runs: list(JavaMicrobenchmarkHarnessObservation)
+ Repetitions of JavaMicrobenchmarkHarnessObservation run.
+
+ """
+ self.name = name
+ self.runs = sorted(runs, key=lambda b: b.value)
+ unit = self.runs[0].unit
+ less_is_better = unit.endswith("/op")
+ values = [b.value for b in self.runs]
+ # Slight kludge to extract the UserCounters for each benchmark
+ self.counters = self.runs[0].counters
+ super().__init__(name, unit, less_is_better, values)
+
+ def __repr__(self):
+ return "JavaMicrobenchmarkHarness[name={},runs={}]".format(
Review comment:
Nit: why include the string "Harness" in all these class names? Seems unnecessary.
##########
File path: dev/archery/archery/cli.py
##########
@@ -450,19 +511,57 @@ def benchmark_run(ctx, rev_or_path, src, preserve, output, cmake_extras,
\b
archery benchmark run --output=run.json
"""
+ if language == "cpp":
+ ctx.forward(benchmark_run_cpp, **kwargs)
+ elif language == "java":
+ ctx.forward(benchmark_run_java, **kwargs)
+
+
+@benchmark.command(name="run_cpp")
+@click.pass_context
+def benchmark_run_cpp(ctx, rev_or_path, src, preserve, output, cmake_extras,
Review comment:
This division also seems unnecessary; CppBenchmarkRunner and JavaBenchmarkRunner seem similar enough to share more code here
##########
File path: dev/archery/archery/cli.py
##########
@@ -367,11 +381,21 @@ def benchmark_common_options(cmd):
click.option("--output", metavar="<output>",
type=click.File("w", encoding="utf8"), default="-",
help="Capture output result into file."),
+ click.option("--language", metavar="<lang>", type=str, default="cpp",
+ show_default=True, callback=check_language,
+ help="Specify target language for the benchmark"),
+ click.option("--build-extras", type=str, multiple=True,
+ help="Extra flags/options to pass to mvn build. "
+ "Can be stackd. For language=java"),
+ click.option("--benchmark-extras", type=str, multiple=True,
+ help="Extra flags/options to pass to mvn benchmark. "
+ "Can be stackd. For language=java"),
Review comment:
```suggestion
"Can be stacked. For language=java"),
```
##########
File path: dev/archery/archery/cli.py
##########
@@ -550,19 +649,34 @@ def benchmark_diff(ctx, src, preserve, output, cmake_extras,
# This should not recompute the benchmark from run.json
archery --quiet benchmark diff WORKSPACE run.json > result.json
"""
+ if language == "cpp":
+ ctx.forward(benchmark_diff_cpp, **kwargs)
+ elif language == "java":
+ ctx.forward(benchmark_diff_java, **kwargs)
+
+
+@benchmark.command(name="diff_cpp")
+@click.pass_context
+def benchmark_diff_cpp(ctx, src, preserve, output, language, cmake_extras,
Review comment:
Again, why separate the languages' benchmarks? It seems especially valuable to have a clearly unified strategy for comparison of benchmarks
##########
File path: dev/archery/archery/benchmark/runner.py
##########
@@ -205,3 +176,133 @@ def suites(self):
continue
yield suite
+
+ @staticmethod
+ def from_rev_or_path(src, root, rev_or_path, cmake_conf, **kwargs):
+ """ Returns a BenchmarkRunner from a path or a git revision.
+
+ First, it checks if `rev_or_path` is a valid path (or string) of a json
+ object that can deserialize to a BenchmarkRunner. If so, it initialize
+ a StaticBenchmarkRunner from it. This allows memoizing the result of a
+ run in a file or a string.
+
+ Second, it checks if `rev_or_path` points to a valid CMake build
+ directory. If so, it creates a CppBenchmarkRunner with this existing
+ CMakeBuild.
+
+ Otherwise, it assumes `rev_or_path` is a revision and clone/checkout
+ the given revision and create a fresh CMakeBuild.
+ """
+ build = None
+ if StaticBenchmarkRunner.is_json_result(rev_or_path):
+ return StaticBenchmarkRunner.from_json(rev_or_path, **kwargs)
+ elif CMakeBuild.is_build_dir(rev_or_path):
+ build = CMakeBuild.from_path(rev_or_path)
+ return CppBenchmarkRunner(build, **kwargs)
+ else:
+ # Revisions can references remote via the `/` character, ensure
+ # that the revision is path friendly
+ path_rev = rev_or_path.replace("/", "_")
+ root_rev = os.path.join(root, path_rev)
+ os.mkdir(root_rev)
+
+ clone_dir = os.path.join(root_rev, "arrow")
+ # Possibly checkout the sources at given revision, no need to
+ # perform cleanup on cloned repository as root_rev is reclaimed.
+ src_rev, _ = src.at_revision(rev_or_path, clone_dir)
+ cmake_def = CppCMakeDefinition(src_rev.cpp, cmake_conf)
+ build_dir = os.path.join(root_rev, "build")
+ return CppBenchmarkRunner(cmake_def.build(build_dir), **kwargs)
+
+
+class JavaBenchmarkRunner(BenchmarkRunner):
+ """ Run suites for Java. """
+
+ def __init__(self, build, **kwargs):
+ """ Initialize a JavaBenchmarkRunner. """
+ self.build = build
+ self.repetitions = 5 # default repetitions for Java
+ super().__init__(**kwargs)
+
+ @staticmethod
+ def default_configuration(**kwargs):
+ """ Returns the default benchmark configuration. """
+ return JavaConfiguration(**kwargs)
+
+ def suite(self, name):
+ """ Returns the resulting benchmarks for a given suite. """
+ # update .m2 directory, which installs target jars
+ self.build.build()
+
+ suite_cmd = JavaMicrobenchmarkHarnessCommand(
+ self.build, self.benchmark_filter)
+
+ # Ensure there will be data
+ benchmark_names = suite_cmd.list_benchmarks()
+ if not benchmark_names:
+ return None
+
+ results = suite_cmd.results(repetitions=self.repetitions)
+ benchmarks = JavaMicrobenchmarkHarness.from_json(results)
+ return BenchmarkSuite(name, benchmarks)
+
+ @property
+ def list_benchmarks(self):
+ """ Returns all suite names """
+ # Ensure build is up-to-date to run benchmarks
+ self.build.build()
+
+ suite_cmd = JavaMicrobenchmarkHarnessCommand(self.build)
+ benchmark_names = suite_cmd.list_benchmarks()
+ for benchmark_name in benchmark_names:
+ yield "{}".format(benchmark_name)
+
+ @property
+ def suites(self):
+ """ Returns all suite for a runner. """
+ suite_name = "JavaBenchmark"
+ suite = self.suite(suite_name)
+
+ # Filter may exclude all benchmarks
+ if not suite:
+ logger.debug("Suite {} executed but no results"
+ .format(suite_name))
+ return
+
+ yield suite
+
+ @staticmethod
+ def from_rev_or_path(src, root, rev_or_path, maven_conf, **kwargs):
+ """ Returns a BenchmarkRunner from a path or a git revision.
+
+ First, it checks if `rev_or_path` is a valid path (or string) of a json
+ object that can deserialize to a BenchmarkRunner. If so, it initialize
+ a StaticBenchmarkRunner from it. This allows memoizing the result of a
+ run in a file or a string.
+
+ Second, it checks if `rev_or_path` points to a valid Maven build
+ directory. If so, it creates a JavaenchmarkRunner with this existing
Review comment:
```suggestion
directory. If so, it creates a JavaBenchmarkRunner with this existing
```
Again, seems like this code didn't need to be repeated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819485881
@ursabot --help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-748568948
ping @liyafan82 @fsaintjacques @kszucs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-697118857
@kiszk Thank you for doing this.
Please note that when running the benchmarks, some flags should be configured properly.
They can be set through environmental variables:
ARROW_ENABLE_UNSAFE_MEMORY_ACCESS = true
ARROW_ENABLE_NULL_CHECK_FOR_GET = false
or through system properties:
arrow.enable_unsafe_memory_access = true
arrow.enable_null_check_for_get = false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-748724392
The Java changes look good to me. However, I am not farmiliar with the archery code. So @fsaintjacques @kszucs could you please take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819486678
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes:
[Scheduled] ursa-i9-9960x: https://conbench.ursa.dev/compare/runs/b04151dd-74b7-4cb5-ac7b-35af58124af7...c28547fd-3722-430f-952e-1b08e321bcaf/
[Scheduled] ursa-thinkcentre-m75q: https://conbench.ursa.dev/compare/runs/0af6683d-a459-4750-868d-e2bbdc5c49dc...761cb228-a197-446a-bfe9-a2834c44ba06/
[Scheduled] ec2-t3-large-us-east-2: https://conbench.ursa.dev/compare/runs/1635513d-25fa-4ec8-9937-b248f6c49ef4...c561174b-c02d-44e9-a6dc-b2a0bd010474/
[Scheduled] ec2-t3-xlarge-us-east-2: https://conbench.ursa.dev/compare/runs/ccf5c2c4-e672-4c37-a511-01a721a8e276...3b4ae039-2fe4-419b-b47a-4dc9a4d0e400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kou closed pull request #8210:
URL: https://github.com/apache/arrow/pull/8210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-757973224
Leaving this for @bkietz , `archery` is not my area of expertise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-725212222
At my end, I can generate the following JSON file by `archery benchmark diff --language=java ...`
@liyafan82 any comments regarding format and parameters are appreciated.
```
benchmark baseline contender change % counters
0 org.apache.arrow.vector.IntBenchmarks.setIntDirectly 22.500 us/op 11.209 us/op -50.182 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 4, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
2 org.apache.arrow.vector.IntBenchmarks.setWithValueHolder 19.031 us/op 6.627 us/op -65.179 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 4, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
1 org.apache.arrow.vector.IntBenchmarks.setWithWriter 32.626 us/op 10.246 us/op -68.594 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 4, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-694335050
https://issues.apache.org/jira/browse/ARROW-10031
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-757763905
I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#discussion_r565051266
##########
File path: dev/archery/archery/benchmark/jmh.py
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from itertools import filterfalse, groupby, tee
+import json
+import subprocess
+from tempfile import NamedTemporaryFile
+
+from .core import Benchmark
+from ..utils.maven import Maven
+
+
+def partition(pred, iterable):
+ # adapted from python's examples
+ t1, t2 = tee(iterable)
+ return list(filter(pred, t1)), list(filterfalse(pred, t2))
+
+
+class JavaMicrobenchmarkHarnessCommand(Maven):
+ """ Run a Java Micro Benchmark Harness
+
+ This assumes the binary supports the standard command line options,
+ notably `--benchmark_filter`, `--benchmark_format`, etc...
+ """
+
+ def __init__(self, build, benchmark_filter=None):
+ self.benchmark_filter = benchmark_filter
+ self.build = build
+ self.maven = Maven()
+
+ def list_benchmarks(self):
+ argv = []
+ if self.benchmark_filter:
+ argv.append("-Dbenchmark.filter={}".format(self.benchmark_filter))
+ result = self.build.list(
+ *argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ """ Extract benchmark names from output. Assume the following output
+ ...
+ Benchmarks:
+ org.apache.arrow.vector.IntBenchmarks.setIntDirectly
+ ...
+ org.apache.arrow.vector.IntBenchmarks.setWithValueHolder
+ org.apache.arrow.vector.IntBenchmarks.setWithWriter
+ ...
+ [INFO]
+ """
+
+ lists = []
+ benchmarks = False
+ for line in str.splitlines(result.stdout.decode("utf-8")):
+ if not benchmarks:
+ if line.startswith("Benchmarks:"):
+ benchmarks = True
+ else:
+ if line.startswith("org.apache.arrow"):
+ lists.append(line)
+ if line.startswith("[INFO]"):
+ benchmarks = False
+ break
+ return lists
+
+ def results(self, repetitions=1):
+ with NamedTemporaryFile(suffix=".json") as out:
+ argv = ["-Dbenchmark.runs={}".format(repetitions),
+ "-Dbenchmark.resultfile={}".format(out.name),
+ "-Dbenchmark.resultformat=json"]
+ if self.benchmark_filter:
+ argv.append(
+ "-Dbenchmark.filter={}".format(self.benchmark_filter)
+ )
+
+ self.build.benchmark(*argv, check=True)
+ return json.load(out)
+
+
+class JavaMicrobenchmarkHarnessObservation:
+ """ Represents one run of a single Java Microbenchmark Harness
+ """
+
+ def __init__(self, benchmark, primaryMetric,
+ forks, warmupIterations, measurementIterations, **counters):
+ self.name = benchmark
+ self.primaryMetric = primaryMetric
+ self.score = primaryMetric["score"]
+ self.scoreUnit = primaryMetric["scoreUnit"]
+ self.forks = forks
+ self.warmups = warmupIterations
+ self.runs = measurementIterations
+ self.counters = {
+ "mode": counters["mode"],
+ "threads": counters["threads"],
+ "warmups": warmupIterations,
+ "warmupTime": counters["warmupTime"],
+ "measurements": measurementIterations,
+ "measurementTime": counters["measurementTime"],
+ "jvmArgs": counters["jvmArgs"]
+ }
+
+ @property
+ def value(self):
+ """ Return the benchmark value."""
+ return self.score
+
+ @property
+ def unit(self):
+ return self.scoreUnit
+
+ def __repr__(self):
+ return str(self.value)
+
+
+class JavaMicrobenchmarkHarness(Benchmark):
+ """ A set of JavaMicrobenchmarkHarnessObservations. """
+
+ def __init__(self, name, runs):
+ """ Initialize a JavaMicrobenchmarkHarness.
+
+ Parameters
+ ----------
+ name: str
+ Name of the benchmark
+ forks: int
+ warmups: int
+ runs: int
+ runs: list(JavaMicrobenchmarkHarnessObservation)
+ Repetitions of JavaMicrobenchmarkHarnessObservation run.
+
+ """
+ self.name = name
+ self.runs = sorted(runs, key=lambda b: b.value)
+ unit = self.runs[0].unit
+ less_is_better = unit.endswith("/op")
+ values = [b.value for b in self.runs]
+ # Slight kludge to extract the UserCounters for each benchmark
+ self.counters = self.runs[0].counters
+ super().__init__(name, unit, less_is_better, values)
+
+ def __repr__(self):
+ return "JavaMicrobenchmarkHarness[name={},runs={}]".format(
Review comment:
[JMH](https://github.com/openjdk/jmh) stands for "Java Microbenchmark Harness". Thus, I used JavaMicrobenchmarkHarness. As you suggested, I will simplify the name as "JavaMicroBench".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-725281667
> At my end, I can generate the following JSON file by `archery benchmark diff --language=java ...`
>
> @liyafan82 any comments regarding format and parameters are appreciated.
>
> ```
> benchmark baseline contender change % counters
> 0 org.apache.arrow.vector.IntBenchmarks.setIntDirectly 22.500 us/op 11.209 us/op -50.182 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 4, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
> 2 org.apache.arrow.vector.IntBenchmarks.setWithValueHolder 19.031 us/op 6.627 us/op -65.179 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 4, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
> 1 org.apache.arrow.vector.IntBenchmarks.setWithWriter 32.626 us/op 10.246 us/op -68.594 {'mode': 'avgt', 'threads': 1, 'warmups': 5, 'warmupTime': '10 s', 'measurements': 4, 'measurementTime': '10 s', 'jvmArgs': ['-Darrow.enable_null_check_for_get=false -Darrow.enable_unsafe_memory_access=true']}
> ```
@kiszk Thanks for your effort. Generally, it looks great! Some minor comments:
1. It is clearer to rename title 'counters' to 'configuration'?
2. I am curious how are the benchmarks sorted, by 'change %'?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-697119568
@liyafan82 Thank you for your comment. I will set these two properties as default for Java benchmarking
.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-large-us-east-2](https://conbench.ursa.dev/compare/runs/d360dee5c8a240df86080fa2e3a80bf1...39375905cbe44997b2d255ff8ec28195/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/138745a3a2dd439885b170555d1dae45...5ae1d12ba4e34a2ebae2977e951f255d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8a3707b919224154a65e5172fd1ae240...2dac391c6c204aaba026d1adafe7ac23/)
[Finished :arrow_down:2.08% :arrow_up:2.14%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/7e94ebe116b24534adcf04237f8063cf...b675c62d35274cadaac2f4511cb82963/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819486678
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes:
[Scheduled] ursa-i9-9960x: https://conbench.ursa.dev/compare/runs/b04151dd-74b7-4cb5-ac7b-35af58124af7...c28547fd-3722-430f-952e-1b08e321bcaf/
[Scheduled] ursa-thinkcentre-m75q: https://conbench.ursa.dev/compare/runs/0af6683d-a459-4750-868d-e2bbdc5c49dc...761cb228-a197-446a-bfe9-a2834c44ba06/
[Finished] ec2-t3-large-us-east-2: https://conbench.ursa.dev/compare/runs/1635513d-25fa-4ec8-9937-b248f6c49ef4...c561174b-c02d-44e9-a6dc-b2a0bd010474/
[Finished] ec2-t3-xlarge-us-east-2: https://conbench.ursa.dev/compare/runs/ccf5c2c4-e672-4c37-a511-01a721a8e276...3b4ae039-2fe4-419b-b47a-4dc9a4d0e400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-726543626
@liyafan82 @fsaintjacques @kszucs Would it be possible to review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
emkornfield edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826247803
Should this be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2](https://conbench.ursa.dev/compare/runs/d360dee5c8a240df86080fa2e3a80bf1...39375905cbe44997b2d255ff8ec28195/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/138745a3a2dd439885b170555d1dae45...5ae1d12ba4e34a2ebae2977e951f255d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8a3707b919224154a65e5172fd1ae240...2dac391c6c204aaba026d1adafe7ac23/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/7e94ebe116b24534adcf04237f8063cf...b675c62d35274cadaac2f4511cb82963/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-828916606
@bkietz @dianaclarke @liyafan82 Would you have additional comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-770455176
@kiszk @bkietz are there more revisions needed here? (it looks like this at least needs a rebase?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819485890
Supported benchmark command examples:
`@ursabot benchmark help`
To run all benchmarks:
`@ursabot please benchmark`
To filter benchmarks by language:
`@ursabot please benchmark lang=Python`
`@ursabot please benchmark lang=C++`
`@ursabot please benchmark lang=R`
To filter Python and R benchmarks by name:
`@ursabot please benchmark name=file-write`
`@ursabot please benchmark name=file-write lang=Python`
`@ursabot please benchmark name=file-.*`
To filter C++ benchmarks by archery --suite-filter and --benchmark-filter:
`@ursabot please benchmark command=cpp-micro --suite-filter=arrow-compute-vector-selection-benchmark --benchmark-filter=TakeStringRandomIndicesWithNulls/262144/2 --iterations=3`
For other `command=cpp-micro` options, please see https://github.com/ursacomputing/benchmarks/blob/main/benchmarks/cpp_micro_benchmarks.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-697119568
@liyafan82 Thank you for your comment. I will set these two properties as default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-725290440
For 1., I will rename the title for cpp and Java.
For 2, you are right as sorted [here](https://github.com/apache/arrow/blob/master/dev/archery/archery/cli.py#L595).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819486678
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes:
[Finished] ursa-i9-9960x: https://conbench.ursa.dev/compare/runs/b04151dd-74b7-4cb5-ac7b-35af58124af7...c28547fd-3722-430f-952e-1b08e321bcaf/
[Finished] ursa-thinkcentre-m75q: https://conbench.ursa.dev/compare/runs/0af6683d-a459-4750-868d-e2bbdc5c49dc...761cb228-a197-446a-bfe9-a2834c44ba06/
[Finished] ec2-t3-large-us-east-2: https://conbench.ursa.dev/compare/runs/1635513d-25fa-4ec8-9937-b248f6c49ef4...c561174b-c02d-44e9-a6dc-b2a0bd010474/
[Finished] ec2-t3-xlarge-us-east-2: https://conbench.ursa.dev/compare/runs/ccf5c2c4-e672-4c37-a511-01a721a8e276...3b4ae039-2fe4-419b-b47a-4dc9a4d0e400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-787525998
CC @dianaclarke who I think has also been working on continuous benchmarking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-697118857
@kiszk Thank you for doing this.
Please note that when running the benchmarks, some flags should be configured properly.
They can be set through environmental variables:
ARROW_ENABLE_UNSAFE_MEMORY_ACCESS = true
ARROW_ENABLE_NULL_CHECK_FOR_GET = false
or through system properties:
arrow.enable_unsafe_memory_access = true
arrow.enable_null_check_for_get = false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-817857640
@emkornfield @bkietz @dianaclarke @liyafan82 I am very sorry for being late to address review comments.
Now, I addressed all of the comments and rebased with the master. Would it be possible to review this PR again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-748916703
Thanks for working on this! At the moment we can only try this out locally since we no longer maintain the buildbot/ursabot setup. I have limited time to review this, so I'd say that if the benchmarking using archery works locally then is should be good to go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#discussion_r522675532
##########
File path: java/performance/pom.xml
##########
@@ -169,10 +173,17 @@
<argument>${benchmark.filter}</argument>
<argument>-f</argument>
<argument>${benchmark.forks}</argument>
+ <argument>-jvmArgs</argument>
+ <argument>${benchmark.jvmargs}</argument>
<argument>-wi</argument>
<argument>${benchmark.warmups}</argument>
<argument>-i</argument>
<argument>${benchmark.runs}</argument>
+ <argument>${benchmark.list}</argument>
+ <argument>-rff</argument>
+ <argument>${benchmark.resultfile}</argument>
+ <argument>-rf</argument>
Review comment:
@liyafan82 Here is a change from #8245 . Since I cannot find to add `-rf json` or not to add it in `<arguments>`, the current implementation always generates `jmh-result.json` and change a file name if it will be overridden.
I would like to update here if there is a selection method to add `-rf json` or not to add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-757765212
@bkietz @xhochy May I ask to review this if possible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2](https://conbench.ursa.dev/compare/runs/d360dee5c8a240df86080fa2e3a80bf1...39375905cbe44997b2d255ff8ec28195/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/138745a3a2dd439885b170555d1dae45...5ae1d12ba4e34a2ebae2977e951f255d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8a3707b919224154a65e5172fd1ae240...2dac391c6c204aaba026d1adafe7ac23/)
[Finished :arrow_down:2.08% :arrow_up:2.14%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/7e94ebe116b24534adcf04237f8063cf...b675c62d35274cadaac2f4511cb82963/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-830492834
Thanks very much for your comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2](https://conbench.ursa.dev/compare/runs/d360dee5c8a240df86080fa2e3a80bf1...39375905cbe44997b2d255ff8ec28195/)
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/138745a3a2dd439885b170555d1dae45...5ae1d12ba4e34a2ebae2977e951f255d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8a3707b919224154a65e5172fd1ae240...2dac391c6c204aaba026d1adafe7ac23/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/7e94ebe116b24534adcf04237f8063cf...b675c62d35274cadaac2f4511cb82963/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819486678
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes:
[Scheduled] ursa-i9-9960x: https://conbench.ursa.dev/compare/runs/b04151dd-74b7-4cb5-ac7b-35af58124af7...c28547fd-3722-430f-952e-1b08e321bcaf/
[Finished] ursa-thinkcentre-m75q: https://conbench.ursa.dev/compare/runs/0af6683d-a459-4750-868d-e2bbdc5c49dc...761cb228-a197-446a-bfe9-a2834c44ba06/
[Finished] ec2-t3-large-us-east-2: https://conbench.ursa.dev/compare/runs/1635513d-25fa-4ec8-9937-b248f6c49ef4...c561174b-c02d-44e9-a6dc-b2a0bd010474/
[Finished] ec2-t3-xlarge-us-east-2: https://conbench.ursa.dev/compare/runs/ccf5c2c4-e672-4c37-a511-01a721a8e276...3b4ae039-2fe4-419b-b47a-4dc9a4d0e400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826247803
Should this be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on a change in pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on a change in pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#discussion_r522824604
##########
File path: java/performance/pom.xml
##########
@@ -169,10 +173,17 @@
<argument>${benchmark.filter}</argument>
<argument>-f</argument>
<argument>${benchmark.forks}</argument>
+ <argument>-jvmArgs</argument>
+ <argument>${benchmark.jvmargs}</argument>
<argument>-wi</argument>
<argument>${benchmark.warmups}</argument>
<argument>-i</argument>
<argument>${benchmark.runs}</argument>
+ <argument>${benchmark.list}</argument>
+ <argument>-rff</argument>
+ <argument>${benchmark.resultfile}</argument>
+ <argument>-rf</argument>
Review comment:
It looks good. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819486678
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes:
[Scheduled] ursa-i9-9960x: https://conbench.ursa.dev/compare/runs/b04151dd-74b7-4cb5-ac7b-35af58124af7...c28547fd-3722-430f-952e-1b08e321bcaf/
[Scheduled] ursa-thinkcentre-m75q: https://conbench.ursa.dev/compare/runs/0af6683d-a459-4750-868d-e2bbdc5c49dc...761cb228-a197-446a-bfe9-a2834c44ba06/
[Scheduled] ec2-t3-large-us-east-2: https://conbench.ursa.dev/compare/runs/1635513d-25fa-4ec8-9937-b248f6c49ef4...c561174b-c02d-44e9-a6dc-b2a0bd010474/
[Finished] ec2-t3-xlarge-us-east-2: https://conbench.ursa.dev/compare/runs/ccf5c2c4-e672-4c37-a511-01a721a8e276...3b4ae039-2fe4-419b-b47a-4dc9a4d0e400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-757763905
I see. Thank you for sharing the latest status.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-828972972
> @bkietz @dianaclarke @liyafan82 Would you have additional comments?
I have no more comments. Hope this PR will be merged soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819493872
I wouldn't be too pedantic here, we can address any issues later. Nice addition, thanks @kiszk!
I submitted a conbench build to check that the archery benchmark command remained compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-770564514
@emkornfield @bkietz Thank you for kind ping. Yes, I need an update since I was swamped last month. I will update this PR this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826247803
Should this be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2](https://conbench.ursa.dev/compare/runs/d360dee5c8a240df86080fa2e3a80bf1...39375905cbe44997b2d255ff8ec28195/)
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/138745a3a2dd439885b170555d1dae45...5ae1d12ba4e34a2ebae2977e951f255d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8a3707b919224154a65e5172fd1ae240...2dac391c6c204aaba026d1adafe7ac23/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/7e94ebe116b24534adcf04237f8063cf...b675c62d35274cadaac2f4511cb82963/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-819486633
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#discussion_r555183556
##########
File path: dev/archery/archery/cli.py
##########
@@ -550,19 +649,34 @@ def benchmark_diff(ctx, src, preserve, output, cmake_extras,
# This should not recompute the benchmark from run.json
archery --quiet benchmark diff WORKSPACE run.json > result.json
"""
+ if language == "cpp":
+ ctx.forward(benchmark_diff_cpp, **kwargs)
+ elif language == "java":
+ ctx.forward(benchmark_diff_java, **kwargs)
+
+
+@benchmark.command(name="diff_cpp")
+@click.pass_context
+def benchmark_diff_cpp(ctx, src, preserve, output, language, cmake_extras,
Review comment:
This separation comes from `@click.pass_context`. To use different options between two languages (e.g. `java-home` or `cxx-flags`), my understanding was that we need different methods with the `@click.pass_context`.
Let me try to have one method with @click.pass_context by lazily parse arguments for each language.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Ursabot
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-697119568
@liyafan82 Thank you for your comment. I will set these two properties as default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
emkornfield edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826247803
Should this be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #8210: ARROW-10031: [CI][Java] Support Java benchmark in Archery
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #8210:
URL: https://github.com/apache/arrow/pull/8210#issuecomment-826248018
Benchmark runs are scheduled for baseline = 5b08205f7e864ed29f53ed3d836845fed62d5d4a and contender = 5d70561ea8d33828b7c5c49d380bafe0e3586225. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-large-us-east-2](https://conbench.ursa.dev/compare/runs/d360dee5c8a240df86080fa2e3a80bf1...39375905cbe44997b2d255ff8ec28195/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/138745a3a2dd439885b170555d1dae45...5ae1d12ba4e34a2ebae2977e951f255d/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8a3707b919224154a65e5172fd1ae240...2dac391c6c204aaba026d1adafe7ac23/)
[Finished :arrow_down:2.08% :arrow_up:2.14%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/7e94ebe116b24534adcf04237f8063cf...b675c62d35274cadaac2f4511cb82963/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org