Re: [DISCUSS] Drop junit-benchmarks

[Jonathan Ellithorpe <jd...@cs.stanford.edu>]

This might be far more heavy-weight than what you are looking for, but I've
been working on implementing the LDBC Social Network Benchmark for TP3:

http://ldbcouncil.org/developer/snb

Jonathan


On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <sp...@gmail.com>
wrote:

> I had long ago built in a model for doing "performance tests" that used:
>
> https://github.com/carrotsearch/junit-benchmarks
>
> I thought we would at some point in the future build those out further but
> that hasn't really happened. I'd probably just not worry about them at this
> point, but while talking to Ted about it, I learned that carrotsearch has
> stopped development on their project and are instead directing folks to use
> JMH:
>
> http://openjdk.java.net/projects/code-tools/jmh/
>
> I think we should just consider dropping the carrotsearch tests in light of
> this - perhaps do a review to see if there are any tests worth moving to
> unit or integration tests.  Then we consider a better model for performance
> testing with JMH (or something else) going forward.
>
> Thoughts?
>

Re: [DISCUSS] Drop junit-benchmarks

[Ted Wilmes <tw...@gmail.com>]

Closing the loop on this guy.  Issue has been entered:
https://issues.apache.org/jira/browse/TINKERPOP-1016

--Ted

On Wed, Dec 2, 2015 at 1:33 PM, Stephen Mallette <sp...@gmail.com>
wrote:

> I like it Ted - sounds like a plan to me.  Not sure how JMH works but maybe
> you can stick with the current basic model that we use with maven:
>
> mvn verify -DskipPerformanceTests=false
>
> if you have to do something else, that's cool.
>
> On Wed, Dec 2, 2015 at 2:25 PM, Ted Wilmes <tw...@gmail.com> wrote:
>
> > I can put a issue in to get JMH setup, remove/port existing tests (we can
> > hash out what gets ported vs. dropped in the issue), and finally add an
> > initial perf test for the addV work.  That sound like a good first step?
> >
> > --Ted
> >
> > On Tue, Dec 1, 2015 at 7:27 PM, Jonathan Ellithorpe <jde@cs.stanford.edu
> >
> > wrote:
> >
> > > Yup :)
> > >
> > > On Tue, Dec 1, 2015 at 3:40 AM Stephen Mallette <sp...@gmail.com>
> > > wrote:
> > >
> > > > Yeah, i think that's a bit too heavy and coarse grained for our
> > purposes.
> > > > I was thinking more in terms of micro-benchmarks for our internal
> use.
> > > >
> > > > Btw, Is your implementation such that any TP3 compliant database
> could
> > > work
> > > > with the LDBC?
> > > >
> > > > On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <
> > > jde@cs.stanford.edu
> > > > >
> > > > wrote:
> > > >
> > > > > This might be far more heavy-weight than what you are looking for,
> > but
> > > > I've
> > > > > been working on implementing the LDBC Social Network Benchmark for
> > TP3:
> > > > >
> > > > > http://ldbcouncil.org/developer/snb
> > > > >
> > > > > Jonathan
> > > > >
> > > > >
> > > > > On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <
> > spmallette@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I had long ago built in a model for doing "performance tests"
> that
> > > > used:
> > > > > >
> > > > > > https://github.com/carrotsearch/junit-benchmarks
> > > > > >
> > > > > > I thought we would at some point in the future build those out
> > > further
> > > > > but
> > > > > > that hasn't really happened. I'd probably just not worry about
> them
> > > at
> > > > > this
> > > > > > point, but while talking to Ted about it, I learned that
> > carrotsearch
> > > > has
> > > > > > stopped development on their project and are instead directing
> > folks
> > > to
> > > > > use
> > > > > > JMH:
> > > > > >
> > > > > > http://openjdk.java.net/projects/code-tools/jmh/
> > > > > >
> > > > > > I think we should just consider dropping the carrotsearch tests
> in
> > > > light
> > > > > of
> > > > > > this - perhaps do a review to see if there are any tests worth
> > moving
> > > > to
> > > > > > unit or integration tests.  Then we consider a better model for
> > > > > performance
> > > > > > testing with JMH (or something else) going forward.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Drop junit-benchmarks

[Stephen Mallette <sp...@gmail.com>]

I like it Ted - sounds like a plan to me.  Not sure how JMH works but maybe
you can stick with the current basic model that we use with maven:

mvn verify -DskipPerformanceTests=false

if you have to do something else, that's cool.

On Wed, Dec 2, 2015 at 2:25 PM, Ted Wilmes <tw...@gmail.com> wrote:

> I can put a issue in to get JMH setup, remove/port existing tests (we can
> hash out what gets ported vs. dropped in the issue), and finally add an
> initial perf test for the addV work.  That sound like a good first step?
>
> --Ted
>
> On Tue, Dec 1, 2015 at 7:27 PM, Jonathan Ellithorpe <jd...@cs.stanford.edu>
> wrote:
>
> > Yup :)
> >
> > On Tue, Dec 1, 2015 at 3:40 AM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > Yeah, i think that's a bit too heavy and coarse grained for our
> purposes.
> > > I was thinking more in terms of micro-benchmarks for our internal use.
> > >
> > > Btw, Is your implementation such that any TP3 compliant database could
> > work
> > > with the LDBC?
> > >
> > > On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <
> > jde@cs.stanford.edu
> > > >
> > > wrote:
> > >
> > > > This might be far more heavy-weight than what you are looking for,
> but
> > > I've
> > > > been working on implementing the LDBC Social Network Benchmark for
> TP3:
> > > >
> > > > http://ldbcouncil.org/developer/snb
> > > >
> > > > Jonathan
> > > >
> > > >
> > > > On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <
> spmallette@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > I had long ago built in a model for doing "performance tests" that
> > > used:
> > > > >
> > > > > https://github.com/carrotsearch/junit-benchmarks
> > > > >
> > > > > I thought we would at some point in the future build those out
> > further
> > > > but
> > > > > that hasn't really happened. I'd probably just not worry about them
> > at
> > > > this
> > > > > point, but while talking to Ted about it, I learned that
> carrotsearch
> > > has
> > > > > stopped development on their project and are instead directing
> folks
> > to
> > > > use
> > > > > JMH:
> > > > >
> > > > > http://openjdk.java.net/projects/code-tools/jmh/
> > > > >
> > > > > I think we should just consider dropping the carrotsearch tests in
> > > light
> > > > of
> > > > > this - perhaps do a review to see if there are any tests worth
> moving
> > > to
> > > > > unit or integration tests.  Then we consider a better model for
> > > > performance
> > > > > testing with JMH (or something else) going forward.
> > > > >
> > > > > Thoughts?
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Drop junit-benchmarks

[Ted Wilmes <tw...@gmail.com>]

I can put a issue in to get JMH setup, remove/port existing tests (we can
hash out what gets ported vs. dropped in the issue), and finally add an
initial perf test for the addV work.  That sound like a good first step?

--Ted

On Tue, Dec 1, 2015 at 7:27 PM, Jonathan Ellithorpe <jd...@cs.stanford.edu>
wrote:

> Yup :)
>
> On Tue, Dec 1, 2015 at 3:40 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > Yeah, i think that's a bit too heavy and coarse grained for our purposes.
> > I was thinking more in terms of micro-benchmarks for our internal use.
> >
> > Btw, Is your implementation such that any TP3 compliant database could
> work
> > with the LDBC?
> >
> > On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <
> jde@cs.stanford.edu
> > >
> > wrote:
> >
> > > This might be far more heavy-weight than what you are looking for, but
> > I've
> > > been working on implementing the LDBC Social Network Benchmark for TP3:
> > >
> > > http://ldbcouncil.org/developer/snb
> > >
> > > Jonathan
> > >
> > >
> > > On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <spmallette@gmail.com
> >
> > > wrote:
> > >
> > > > I had long ago built in a model for doing "performance tests" that
> > used:
> > > >
> > > > https://github.com/carrotsearch/junit-benchmarks
> > > >
> > > > I thought we would at some point in the future build those out
> further
> > > but
> > > > that hasn't really happened. I'd probably just not worry about them
> at
> > > this
> > > > point, but while talking to Ted about it, I learned that carrotsearch
> > has
> > > > stopped development on their project and are instead directing folks
> to
> > > use
> > > > JMH:
> > > >
> > > > http://openjdk.java.net/projects/code-tools/jmh/
> > > >
> > > > I think we should just consider dropping the carrotsearch tests in
> > light
> > > of
> > > > this - perhaps do a review to see if there are any tests worth moving
> > to
> > > > unit or integration tests.  Then we consider a better model for
> > > performance
> > > > testing with JMH (or something else) going forward.
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>

Re: [DISCUSS] Drop junit-benchmarks

[Jonathan Ellithorpe <jd...@cs.stanford.edu>]

Yup :)

On Tue, Dec 1, 2015 at 3:40 AM Stephen Mallette <sp...@gmail.com>
wrote:

> Yeah, i think that's a bit too heavy and coarse grained for our purposes.
> I was thinking more in terms of micro-benchmarks for our internal use.
>
> Btw, Is your implementation such that any TP3 compliant database could work
> with the LDBC?
>
> On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <jde@cs.stanford.edu
> >
> wrote:
>
> > This might be far more heavy-weight than what you are looking for, but
> I've
> > been working on implementing the LDBC Social Network Benchmark for TP3:
> >
> > http://ldbcouncil.org/developer/snb
> >
> > Jonathan
> >
> >
> > On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > I had long ago built in a model for doing "performance tests" that
> used:
> > >
> > > https://github.com/carrotsearch/junit-benchmarks
> > >
> > > I thought we would at some point in the future build those out further
> > but
> > > that hasn't really happened. I'd probably just not worry about them at
> > this
> > > point, but while talking to Ted about it, I learned that carrotsearch
> has
> > > stopped development on their project and are instead directing folks to
> > use
> > > JMH:
> > >
> > > http://openjdk.java.net/projects/code-tools/jmh/
> > >
> > > I think we should just consider dropping the carrotsearch tests in
> light
> > of
> > > this - perhaps do a review to see if there are any tests worth moving
> to
> > > unit or integration tests.  Then we consider a better model for
> > performance
> > > testing with JMH (or something else) going forward.
> > >
> > > Thoughts?
> > >
> >
>

Re: [DISCUSS] Drop junit-benchmarks

[Stephen Mallette <sp...@gmail.com>]

We haven't run tests between releases precisely because it is "tricky
getting usable numbers for comparison".  the existing tests were fairly
arbitrary in how they were selected - i was mostly trying to develop a
model we could follow.  i wasn't really trying too hard to do much else.

> To start out, I'm thinking there could at least be a test for each new
performance fix going forward unless that fix hits on an area that already
has performance coverage.

that sounds sensible to me

On Tue, Dec 1, 2015 at 9:43 AM, Ted Wilmes <tw...@gmail.com> wrote:

> I think the move to JMH is a good idea.  How have you guys been using the
> existing tests? For example, have you generated the reports at various
> points and then done comparisons between releases?  It would be neat to
> have the tests run along with the Travis build but it could be tricky
> getting usable numbers for comparison between builds since we don't have
> any control over that box (other processes running, etc.).  I think there
> could be some benefit even initially if we converted all or some set of the
> existing tests, and then did a build on a known box (one of our own) around
> code freeze time to see if we introduced any new, unintended issues since
> the last release.  We'd keep all the reports as we did these for comparison
> purposes and at the same time, work towards more automation.  Jenkins has a
> pretty slick looking JMH plugin that helps you track performance trends and
> detect regressions. [1]  I haven't found one for Travis yet, but the JMH
> output is pretty straightforward so it might not be that hard to put one
> together.
>
> I think the other issue is how do we want to build the performance tests
> out from a coverage standpoint?  I don't think it would be too much trouble
> to convert the existing tests to JMH but Stephen, I like your idea of
> evaluating them first to see if we'd like to keep/tweak them.  To start
> out, I'm thinking there could at least be a test for each new performance
> fix going forward unless that fix hits on an area that already has
> performance coverage.  For example, I would to add one for the
> addV/addVertex work.  This would help us track areas of particular interest
> from release-to-release and eventually, build-to-build.
>
> --Ted
>
> [1] https://github.com/blackboard/jmh-jenkins
>
>
> On Tue, Dec 1, 2015 at 5:40 AM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > Yeah, i think that's a bit too heavy and coarse grained for our purposes.
> > I was thinking more in terms of micro-benchmarks for our internal use.
> >
> > Btw, Is your implementation such that any TP3 compliant database could
> work
> > with the LDBC?
> >
> > On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <
> jde@cs.stanford.edu
> > >
> > wrote:
> >
> > > This might be far more heavy-weight than what you are looking for, but
> > I've
> > > been working on implementing the LDBC Social Network Benchmark for TP3:
> > >
> > > http://ldbcouncil.org/developer/snb
> > >
> > > Jonathan
> > >
> > >
> > > On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <spmallette@gmail.com
> >
> > > wrote:
> > >
> > > > I had long ago built in a model for doing "performance tests" that
> > used:
> > > >
> > > > https://github.com/carrotsearch/junit-benchmarks
> > > >
> > > > I thought we would at some point in the future build those out
> further
> > > but
> > > > that hasn't really happened. I'd probably just not worry about them
> at
> > > this
> > > > point, but while talking to Ted about it, I learned that carrotsearch
> > has
> > > > stopped development on their project and are instead directing folks
> to
> > > use
> > > > JMH:
> > > >
> > > > http://openjdk.java.net/projects/code-tools/jmh/
> > > >
> > > > I think we should just consider dropping the carrotsearch tests in
> > light
> > > of
> > > > this - perhaps do a review to see if there are any tests worth moving
> > to
> > > > unit or integration tests.  Then we consider a better model for
> > > performance
> > > > testing with JMH (or something else) going forward.
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>

Re: [DISCUSS] Drop junit-benchmarks

[Ted Wilmes <tw...@gmail.com>]

I think the move to JMH is a good idea.  How have you guys been using the
existing tests? For example, have you generated the reports at various
points and then done comparisons between releases?  It would be neat to
have the tests run along with the Travis build but it could be tricky
getting usable numbers for comparison between builds since we don't have
any control over that box (other processes running, etc.).  I think there
could be some benefit even initially if we converted all or some set of the
existing tests, and then did a build on a known box (one of our own) around
code freeze time to see if we introduced any new, unintended issues since
the last release.  We'd keep all the reports as we did these for comparison
purposes and at the same time, work towards more automation.  Jenkins has a
pretty slick looking JMH plugin that helps you track performance trends and
detect regressions. [1]  I haven't found one for Travis yet, but the JMH
output is pretty straightforward so it might not be that hard to put one
together.

I think the other issue is how do we want to build the performance tests
out from a coverage standpoint?  I don't think it would be too much trouble
to convert the existing tests to JMH but Stephen, I like your idea of
evaluating them first to see if we'd like to keep/tweak them.  To start
out, I'm thinking there could at least be a test for each new performance
fix going forward unless that fix hits on an area that already has
performance coverage.  For example, I would to add one for the
addV/addVertex work.  This would help us track areas of particular interest
from release-to-release and eventually, build-to-build.

--Ted

[1] https://github.com/blackboard/jmh-jenkins


On Tue, Dec 1, 2015 at 5:40 AM, Stephen Mallette <sp...@gmail.com>
wrote:

> Yeah, i think that's a bit too heavy and coarse grained for our purposes.
> I was thinking more in terms of micro-benchmarks for our internal use.
>
> Btw, Is your implementation such that any TP3 compliant database could work
> with the LDBC?
>
> On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <jde@cs.stanford.edu
> >
> wrote:
>
> > This might be far more heavy-weight than what you are looking for, but
> I've
> > been working on implementing the LDBC Social Network Benchmark for TP3:
> >
> > http://ldbcouncil.org/developer/snb
> >
> > Jonathan
> >
> >
> > On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > I had long ago built in a model for doing "performance tests" that
> used:
> > >
> > > https://github.com/carrotsearch/junit-benchmarks
> > >
> > > I thought we would at some point in the future build those out further
> > but
> > > that hasn't really happened. I'd probably just not worry about them at
> > this
> > > point, but while talking to Ted about it, I learned that carrotsearch
> has
> > > stopped development on their project and are instead directing folks to
> > use
> > > JMH:
> > >
> > > http://openjdk.java.net/projects/code-tools/jmh/
> > >
> > > I think we should just consider dropping the carrotsearch tests in
> light
> > of
> > > this - perhaps do a review to see if there are any tests worth moving
> to
> > > unit or integration tests.  Then we consider a better model for
> > performance
> > > testing with JMH (or something else) going forward.
> > >
> > > Thoughts?
> > >
> >
>

Re: [DISCUSS] Drop junit-benchmarks

[Stephen Mallette <sp...@gmail.com>]

Yeah, i think that's a bit too heavy and coarse grained for our purposes.
I was thinking more in terms of micro-benchmarks for our internal use.

Btw, Is your implementation such that any TP3 compliant database could work
with the LDBC?

On Mon, Nov 30, 2015 at 11:49 PM, Jonathan Ellithorpe <jd...@cs.stanford.edu>
wrote:

> This might be far more heavy-weight than what you are looking for, but I've
> been working on implementing the LDBC Social Network Benchmark for TP3:
>
> http://ldbcouncil.org/developer/snb
>
> Jonathan
>
>
> On Mon, Nov 30, 2015 at 8:28 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > I had long ago built in a model for doing "performance tests" that used:
> >
> > https://github.com/carrotsearch/junit-benchmarks
> >
> > I thought we would at some point in the future build those out further
> but
> > that hasn't really happened. I'd probably just not worry about them at
> this
> > point, but while talking to Ted about it, I learned that carrotsearch has
> > stopped development on their project and are instead directing folks to
> use
> > JMH:
> >
> > http://openjdk.java.net/projects/code-tools/jmh/
> >
> > I think we should just consider dropping the carrotsearch tests in light
> of
> > this - perhaps do a review to see if there are any tests worth moving to
> > unit or integration tests.  Then we consider a better model for
> performance
> > testing with JMH (or something else) going forward.
> >
> > Thoughts?
> >
>