You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by twilmes <gi...@git.apache.org> on 2016/03/21 22:55:33 UTC
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
GitHub user twilmes opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/274
TINKERPOP-1016 Replace junit-benchmarks with JMH
This PR includes the following:
- deprecates JUnit benchmark performance tests and added notes to the 3.2.0 upgrade docs (the deprecation vs. outright deletion may be overkill but I thought that some providers may still be using these)
- adds a new JMH gremlin-benchmark module with basic starter benchmarks. These benchmarks are meant to be used for internal TinkerPop benchmarking purposes as opposed to provider benchmarking
- added a `Developing Benchmarks` section to the developer docs
You can run the benchmarks both from your IDE and maven. They are nested inside of a JUnit test runner. In addition, a benchmark uber-jar can be built and benchmarks can be run using the built-in JMH test runner:
`java -jar gremlin-benchmark-3.2.0-SNAPSHOT.jar`
By default, benchmark output is stored in JSON format in a benchmark directory. To run the benchmarks from maven do `mvn test -DskipBenchmarks=false`.
`mvn clean install` and `./processDocs.sh --dryRun` checked out ok.
VOTE: +1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-tinkerpop TINKERPOP-1016
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/274.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #274
----
commit 154a12b1c8be1052c0ed56e4a7e28032da5e5bc5
Author: Ted Wilmes <tw...@gmail.com>
Date: 2016-03-21T18:22:09Z
This commit includes the following:
* added new gremlin-benchmark module and initial benchmark classes
* deprecated the JUnit benchmark classes, noted this deprecation in upgrade doc
* expanded scope of LoadGraphWith annotation to allow for class level annotation
* added developer benchmarking documentation
commit cd4d9ccc9f0d46e20fc441fb39f85b53ac8eebff
Author: Ted Wilmes <tw...@gmail.com>
Date: 2016-03-21T21:19:48Z
Added gremlin-benchmark pom.xml to module and updated javadocs.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-200439071
Not sure if you did this already but, when you deprecate you should always remember to add a JIRA ticket to remove the deprecation in the future. Here's an example:
https://issues.apache.org/jira/browse/TINKERPOP-1094
Note the "breaking" and "deprecation" tags. I don't think there's many folks using the performance tests, but probably makes sense to do what you did there. No sense breaking provider's implementations over this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-200434586
If it's just overwriting on every run, then putting in `/target` would be better. If you want to update your PR to append a timestamp or something to what's in `/benchmarks` then I'd say keep it there as that would work better for comparisons.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-199914657
This looks really good. Its pretty simple as it stands, but a good foundation for us to build on. Already have a list of JIRA features to add once this gets merged to master.
VOTE +1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by twilmes <gi...@git.apache.org>.
Github user twilmes commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-201332229
Pushed updates that disabled the creation of the `dependency-reduced-pom.xml` and I upped the default heap on `AbstractBenchmarkBase` to 1G.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-201265647
I took an error while running the benchmarks:
```text
# JMH 1.11.3 (released 71 days ago)
# VM version: JDK 1.8.0_72, VM 25.72-b15
# VM invoker: /usr/lib/jvm/java-8-oracle/jre/bin/java
# VM options: -server -Xms512m -Xmx512m
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.tinkerpop.gremlin.process.GraphTraversalBenchmark.g_V_out_out_out_path
# Run progress: 72.73% complete, ETA 00:03:01
# Fork: 1 of 2
# Warmup Iteration 1: <failure>
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.LinkedHashMap$LinkedKeySet.iterator(LinkedHashMap.java:539)
at java.util.HashSet.iterator(HashSet.java:172)
at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1039)
at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)
at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.ImmutablePath.<init>(ImmutablePath.java:58)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.ImmutablePath.extend(ImmutablePath.java:68)
at org.apache.tinkerpop.gremlin.process.traversal.traverser.B_LP_O_P_S_SE_SL_Traverser.split(B_LP_O_P_S_SE_SL_Traverser.java:65)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:44)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:140)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.MapStep.processNextStart(MapStep.java:36)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.fill(Traversal.java:146)
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.toList(Traversal.java:103)
at org.apache.tinkerpop.gremlin.process.GraphTraversalBenchmark.g_V_out_out_out_path(GraphTraversalBenchmark.java:56)
at org.apache.tinkerpop.gremlin.process.generated.GraphTraversalBenchmark_g_V_out_out_out_path_jmhTest.g_V_out_out_out_path_thrpt_jmhStub(GraphTraversalBenchmark_g_V_out_out_out_path_jmhTest.java:104)
at org.apache.tinkerpop.gremlin.process.generated.GraphTraversalBenchmark_g_V_out_out_out_path_jmhTest.g_V_out_out_out_path_Throughput(GraphTraversalBenchmark_g_V_out_out_out_path_jmhTest.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:430)
at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:412)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
is that expected in some way (it seemed to keep running)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by twilmes <gi...@git.apache.org>.
Github user twilmes closed the pull request at:
https://github.com/apache/incubator-tinkerpop/pull/274
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-202656792
VOTE +1 - benchmarks ran for me without issue now. Not a big deal, but I noticed that i have to be in the gremlin-benchmark directory to do: `mvn test -DskipBenchmarks=false` or else i get an error which makes sense because gremlin-shaded won't build properly on the "test" target. Anyway, maybe something to document after merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by twilmes <gi...@git.apache.org>.
Github user twilmes commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-200309448
I had done it like that so you could switch back and forth between branches, run benchmarks, and the results would be persistent for comparison purposes without copying them out of `target`. Having said that, I'm just using the benchmark class name as the name of the file so these would get overwritten every run anyway. I could update that naming to include an incrementing number in the file name. I'm not opposed to putting them under target, though. I'd imagine the usual performance tuning cycle may go something like iterate on perf improvements in your branch, running benchmarks as you make updates. Then commit, switch back to master or latest stable, run benchmarks again, and note how much of an improvement you made.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-200297011
@twilmes is there any reason to write to `./benchmarks` rather than the build directory like `target/benchmarks`? I think it would be better to keep all build related things there if possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-201267112
So benchmarks complete and then i get this file added:
```text
$ git status
On branch TINKERPOP-1016
Your branch is up-to-date with 'origin/TINKERPOP-1016'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
gremlin-benchmark/dependency-reduced-pom.xml
```
I think it came as a result of the benchmark run (i certainly didn't add it myself). Should that be in .gitignore?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by twilmes <gi...@git.apache.org>.
Github user twilmes commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-200546140
Thanks for the reminder. I had not entered a ticket for the deprecation. TINKERPOP-1235 has been added to cover this. I have also pushed a commit that appends the current timestamp in yyyyMMddHHmmss format to the benchmark result files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1016 Replace junit-ben...
Posted by twilmes <gi...@git.apache.org>.
Github user twilmes commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/274#issuecomment-202866335
I accidentally overwrote the merge comment so this didn't auto-close. It has been merged into master commit 682d313e8632c10470f0376f38c169acb6e047c3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---