[jira] [Commented] (TINKERPOP3-231) Expand the PerformanceSuite

["Daniel Kuppitz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP3-231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526835#comment-14526835 ] 

Daniel Kuppitz commented on TINKERPOP3-231:
-------------------------------------------

I created a separate project that...

a) spawns a Docker new container
b) checks out the current master and
c) records the performance of a bunch of queries

What I found was that the results are pretty worthless. Differences in execution times are mostly (or always?) just noise. In my opinion TinkerGraph is too fast - automated benchmarks between releases / commits can lead to a lot of wrong assumptions. You can even get totally different results if a commit contains only changes in the docs, which wouldn't make any sense at all.

> Expand the PerformanceSuite
> ---------------------------
>
>                 Key: TINKERPOP3-231
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP3-231
>             Project: TinkerPop 3
>          Issue Type: Improvement
>          Components: test-suite
>            Reporter: stephen mallette
>            Assignee: Daniel Kuppitz
>             Fix For: 3.0.0.GA
>
>
> We current have:
> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-test/src/main/java/com/tinkerpop/gremlin/structure/StructurePerformanceSuite.java
> It doesn't do much - basic/simple tests for simple reads/writes.  We should build this out for "traversal" oriented tests.  Not sure if we want to have a {{ProcessPerformanceSuite}} for that or not.  The model would be the same.  
> This would be nice because it's then easy to check for performance regressions between branches, releases, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)