You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by Jeff Hagelberg <jn...@us.ibm.com> on 2016/12/12 23:14:17 UTC

Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.

The code being submitted has only been tested with TP 3 at this point. There are some issues that need to be fixed before this can be used with TP 2:

    * FunctionGenerator - generates functions that take GraphTraversal as an argument. That should be changed to GreminPipeline when using TP 2. The return type should probably come from a new method in GremlinExpressionFactory
    * loop support - for TP2, we will need to add logic to make sure that the loop call is not separated from the alias it references
    * alias handling - the alias safe/restore logic in ExpandOrsOptimization.getBaseResultExpression() may be need to adjusted for TP2
    * possibly other issues, TBD


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java b60edef998b4c8361b4f194539577014428483db 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties eea46a6abf3e8f0e1c81cd62e765d8c72e8fa623 
  pom.xml bdfd785194e35239db5feab93a5fa70ef20f84fb 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 6b6896150efe185cf4f2cb4ae005bd5aaa0b359a 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java ca1ad5df9fc01b5c50c2dcc31098beca1cdfb76d 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 3677544d73183116cd0bea4a1b04be11f19b5c1a 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java 1a3faf778f01582e476792780a9dbaa8a4010d29 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 84608d9db54058ae59595acc42710b23c0c9b918 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a270b9789d67cb124e7929d495da04878f2ea35d 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala a61ff982915684b72fe7067eea2fc2ac75d31baf 
  repository/src/test/java/org/apache/atlas/repository/graph/GremlinQueryOptimizerTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/#review163818
-----------------------------------------------------------



Any other comments on this?  If not, I will commit it this afternoon.

- Jeff Hagelberg


On Jan. 31, 2017, 10:54 p.m., Jeff Hagelberg wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54680/
> -----------------------------------------------------------
> 
> (Updated Jan. 31, 2017, 10:54 p.m.)
> 
> 
> Review request for atlas and David Kantor.
> 
> 
> Bugs: ATLAS-1369
>     https://issues.apache.org/jira/browse/ATLAS-1369
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.
> 
> 
> Major changes:
> 
>     * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
>     * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
>     * Refactoring of Groovy Expression classes
>         * new - AbstractFunctionExpression - super class of anything with a caller
>         * added 4 new methods to GroovyExpression
>         * now, each GroovyExpression is a TraversalStepType associated with it
>         These are used by the optimization framework
> 
>     * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.
> 
>     * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
>     * 
> Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).
> 
> Please review these changes.  Silence implies consent.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
>   common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
>   common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
>   common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
>   common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
>   common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
>   common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
>   common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
>   common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
>   common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
>   common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
>   common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
>   common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
>   common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
>   common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
>   common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
>   common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
>   common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
>   common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
>   distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
>   graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
>   intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
>   pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
>   repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
>   repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
>   repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
>   repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
>   repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
>   repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
>   repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 
> 
> Diff: https://reviews.apache.org/r/54680/diff/
> 
> 
> Testing
> -------
> 
> Ran full Atlas build, no regresssions found.
> 
> 
> Thanks,
> 
> Jeff Hagelberg
> 
>

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Feb. 2, 2017, 7:48 p.m.)


Review request for atlas and David Kantor.


Changes
-------

apply fix for lineage tests in webapp


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
    * 
Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).

Please review these changes.  Silence implies consent.


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java 4db4773a92f1627d60de17aba9e9fac824499bf3 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java 2848a20ebe40b2670cf2479f90e8a3dd29883449 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 1590aeea3d8d9179d53e3f968125b139c46a2946 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 
  shaded/hbase-client-shaded/pom.xml 86da66094a3537519f4d46eaba655f14e61dd4ae 
  shaded/hbase-server-shaded/pom.xml d8b04b00fc067cf4b85031f0f902f25da539f0f1 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Feb. 2, 2017, 6:04 p.m.)


Review request for atlas and David Kantor.


Changes
-------

Restored Java 7 compatibility, fixed HBase test failures in repository.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
    * 
Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).

Please review these changes.  Silence implies consent.


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java 2848a20ebe40b2670cf2479f90e8a3dd29883449 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 1590aeea3d8d9179d53e3f968125b139c46a2946 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 
  shaded/hbase-client-shaded/pom.xml 86da66094a3537519f4d46eaba655f14e61dd4ae 
  shaded/hbase-server-shaded/pom.xml d8b04b00fc067cf4b85031f0f902f25da539f0f1 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 31, 2017, 10:54 p.m.)


Review request for atlas and David Kantor.


Changes
-------

Rebased, addressed review comments.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
    * 
Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).

Please review these changes.  Silence implies consent.


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.


> On Jan. 31, 2017, 6:43 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java, line 43
> > <https://reviews.apache.org/r/54680/diff/5/?file=1619556#file1619556line43>
> >
> >     Is it Gremlin that does not use the indices, or is it Titan?  I thought that was a Titan bug.

Well, it's a little of both.  Gremlin provides a framework which Titan implements.  I guess it is more the responsibility of Titan to execute the queries efficiently though.


> On Jan. 31, 2017, 6:43 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java, lines 158-161
> > <https://reviews.apache.org/r/54680/diff/5/?file=1619572#file1619572line158>
> >
> >     I think is comment is obsolete - a holdover from the original superTypes optimization - and should be removed.

Good catch, thanks!


- Jeff


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/#review163667
-----------------------------------------------------------


On Jan. 31, 2017, 10:54 p.m., Jeff Hagelberg wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54680/
> -----------------------------------------------------------
> 
> (Updated Jan. 31, 2017, 10:54 p.m.)
> 
> 
> Review request for atlas and David Kantor.
> 
> 
> Bugs: ATLAS-1369
>     https://issues.apache.org/jira/browse/ATLAS-1369
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.
> 
> 
> Major changes:
> 
>     * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
>     * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
>     * Refactoring of Groovy Expression classes
>         * new - AbstractFunctionExpression - super class of anything with a caller
>         * added 4 new methods to GroovyExpression
>         * now, each GroovyExpression is a TraversalStepType associated with it
>         These are used by the optimization framework
> 
>     * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.
> 
>     * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
>     * 
> Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).
> 
> Please review these changes.  Silence implies consent.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
>   common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
>   common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
>   common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
>   common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
>   common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
>   common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
>   common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
>   common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
>   common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
>   common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
>   common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
>   common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
>   common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
>   common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
>   common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
>   common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
>   common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
>   common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
>   distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
>   graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
>   intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
>   pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
>   repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
>   repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
>   repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
>   repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
>   repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
>   repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
>   repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 
> 
> Diff: https://reviews.apache.org/r/54680/diff/
> 
> 
> Testing
> -------
> 
> Ran full Atlas build, no regresssions found.
> 
> 
> Thanks,
> 
> Jeff Hagelberg
> 
>

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by David Kantor <dk...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/#review163667
-----------------------------------------------------------


Fix it, then Ship it!




Fix it then ship it


common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java (line 35)
<https://reviews.apache.org/r/54680/#comment235141>

    Consider renaming to VariableDeclaration, no need to abbreviate the name.



common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java (line 25)
<https://reviews.apache.org/r/54680/#comment235140>

    typo



repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java (line 185)
<https://reviews.apache.org/r/54680/#comment235155>

    remove commented-out code



repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java (lines 291 - 292)
<https://reviews.apache.org/r/54680/#comment235157>

    Rename to startIndex and endIndex



repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java (line 43)
<https://reviews.apache.org/r/54680/#comment235158>

    Is it Gremlin that does not use the indices, or is it Titan?  I thought that was a Titan bug.



repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java (lines 34 - 35)
<https://reviews.apache.org/r/54680/#comment235161>

    It's the GremlinOptimization implementations that apply the specific optimizations.  This javadoc implies that GremlinQueryOptimizer is doing it.  Perhaps it makes sense to have the details of each optimization documented in the implementation classes, and change this javadoc to be more general.



repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java (lines 40 - 42)
<https://reviews.apache.org/r/54680/#comment235162>

    Remove empty comment



repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java (line 25)
<https://reviews.apache.org/r/54680/#comment235163>

    Incorrect javadoc, should mention path expressions not aliases.



repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java (lines 157 - 160)
<https://reviews.apache.org/r/54680/#comment235166>

    I think is comment is obsolete - a holdover from the original superTypes optimization - and should be removed.


- David Kantor


On Jan. 31, 2017, 2:11 p.m., Jeff Hagelberg wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54680/
> -----------------------------------------------------------
> 
> (Updated Jan. 31, 2017, 2:11 p.m.)
> 
> 
> Review request for atlas and David Kantor.
> 
> 
> Bugs: ATLAS-1369
>     https://issues.apache.org/jira/browse/ATLAS-1369
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.
> 
> 
> Major changes:
> 
>     * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
>     * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
>     * Refactoring of Groovy Expression classes
>         * new - AbstractFunctionExpression - super class of anything with a caller
>         * added 4 new methods to GroovyExpression
>         * now, each GroovyExpression is a TraversalStepType associated with it
>         These are used by the optimization framework
> 
>     * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.
> 
>     * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
>     * 
> Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).
> 
> Please review these changes.  Silence implies consent.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
>   common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
>   common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
>   common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
>   common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
>   common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
>   common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
>   common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
>   common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
>   common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
>   common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
>   common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
>   common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
>   common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
>   common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
>   common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
>   common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
>   common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
>   common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
>   distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
>   graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
>   intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
>   pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
>   repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
>   repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
>   repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
>   repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
>   repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
>   repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
>   repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/util/CompiledQueryCacheKeyTest.java c926f4d7d793ef095b20853277f3ba979483e039 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 
> 
> Diff: https://reviews.apache.org/r/54680/diff/
> 
> 
> Testing
> -------
> 
> Ran full Atlas build, no regresssions found.
> 
> 
> Thanks,
> 
> Jeff Hagelberg
> 
>

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 31, 2017, 2:11 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
    * 
Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).

Please review these changes.  Silence implies consent.


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/util/CompiledQueryCacheKeyTest.java c926f4d7d793ef095b20853277f3ba979483e039 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 11:47 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization
    * 
Note - with these changes we saw a very significant improvement when using IBM Graph as the Atlas Backend (which uses Titan 1 under the covers).


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/util/CompiledQueryCacheKeyTest.java c926f4d7d793ef095b20853277f3ba979483e039 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 11:18 p.m.)


Review request for atlas and David Kantor.


Changes
-------

Cleaned up trailing whitespace.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/util/CompiledQueryCacheKeyTest.java c926f4d7d793ef095b20853277f3ba979483e039 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 11:09 p.m.)


Review request for atlas and David Kantor.


Changes
-------

Rebased changes.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties d9e2f6efae611b1360d1a14ecabc981247f2a5b8 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml a98579276b7d941e431b35594e363e07fee4efc0 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a04dd95d01d6faa4beeadb042553ba2028a9f1eb 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/util/CompiledQueryCacheKeyTest.java c926f4d7d793ef095b20853277f3ba979483e039 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 5:23 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing (updated)
-------

Ran full Atlas build, no regresssions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 5:04 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 4:59 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework
    * Bumped up the version of Guava to get Lists.cartesianProduct

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 4:58 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework
    * Bumped up the version of Guava to get Lists.caretesianProduct

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 4:46 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 4:29 p.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off.  It is provided as a fallback in the event that issues are found with the optimizer.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Jan. 30, 2017, 4:26 p.m.)


Review request for atlas and David Kantor.


Changes
-------

Added support for Gremlin 2.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.

The code being submitted has only been tested with TP 3 at this point. There are some issues that need to be fixed before this can be used with TP 2:

    * FunctionGenerator - generates functions that take GraphTraversal as an argument. That should be changed to GreminPipeline when using TP 2. The return type should probably come from a new method in GremlinExpressionFactory
    * loop support - for TP2, we will need to add logic to make sure that the loop call is not separated from the alias it references
    * alias handling - the alias safe/restore logic in ExpandOrsOptimization.getBaseResultExpression() may be need to adjusted for TP2
    * possibly other issues, TBD


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off when using TP 3. It is always off in TP2.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java dd9b1d58e944a4c172a571f4dcacc6601bce22be 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties 303ce7b00c93f2499e7b18c5f3a131b50c150b69 
  graphdb/titan0/pom.xml 58a5cb852ad9d1e78fc8ced0b153c39d7d816132 
  intg/pom.xml 52b5ef544c5ad5f97ef78fd2c8eb979f5e2a7b6d 
  pom.xml 0321d8462aeeb4bdd2ef593e22ef1b6dbf442bc1 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 18587391dc8d384d29e069e04a3bf787d5865452 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java e862769b7674449a2bebc8f35fdfa07e310eeb88 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 6c326b2ea9310fde9515dd98afbb8f97f73b8888 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OrderFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/PathExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/RangeFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java be02891cad3020e50d5244fd2aaf3c4af3a5954f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 7b2b753b930467e015d364819c5baa983fce0cf8 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 6655085ee11c94addca1564cf77bbdb001c4586f 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala f7ba71a444ed327a03c061520646c868c0ba4814 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java f2ca6a8fb1573755d16497236358358a07881e9c 
  repository/src/test/java/org/apache/atlas/repository/graph/AbstractGremlinQueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin2QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/Gremlin3QueryOptimizerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/graph/TestIntSequence.java PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 22717ff7433958142dbf126a9bd6e72bc9151d12 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg

Re: Review Request 54680: ATLAS-1369 - Optimize gremlin queries generated by DSL translator

Posted by Jeff Hagelberg <jn...@us.ibm.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54680/
-----------------------------------------------------------

(Updated Dec. 13, 2016, 12:09 a.m.)


Review request for atlas and David Kantor.


Bugs: ATLAS-1369
    https://issues.apache.org/jira/browse/ATLAS-1369


Repository: atlas


Description (updated)
-------

The way the queries are generated now, in many cases the indices are not used when executing queries. The main problem is that gremlin cannot currently use "has" expressions within an "and" or an "or". This JIRA introduces a GremlinQueryOptimizer that attempts to extract "has" expressions out of "and" expressions and turn "or" expressions into unions.

The code being submitted has only been tested with TP 3 at this point. There are some issues that need to be fixed before this can be used with TP 2:

    * FunctionGenerator - generates functions that take GraphTraversal as an argument. That should be changed to GreminPipeline when using TP 2. The return type should probably come from a new method in GremlinExpressionFactory
    * loop support - for TP2, we will need to add logic to make sure that the loop call is not separated from the alias it references
    * alias handling - the alias safe/restore logic in ExpandOrsOptimization.getBaseResultExpression() may be need to adjusted for TP2
    * possibly other issues, TBD


Major changes:

    * Updated version of Guava used by Atlas, so that could use Lists.cartesianProduct
    * New package - org.apache.atlas.gremlin.optimizer - contains optimization framework
    * Refactoring of Groovy Expression classes
        * new - AbstractFunctionExpression - super class of anything with a caller
        * added 4 new methods to GroovyExpression
        * now, each GroovyExpression is a TraversalStepType associated with it
        These are used by the optimization framework

    * Added feature toggle : atlas.query.gremlinOptimizerEnabled. The feature turns optimization on/off when using TP 3. It is always off in TP2.

    * Most of the new logic is in GremlinQueryOptimizer, ExpandAndsOptimization, and ExpandOrsOptimization


Diffs
-----

  common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java 49eaae87e593d4e350ecb9a51a83e967d95b514b 
  common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java 0aec5d03ea6e2a108129b20003d390e826f76bb5 
  common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ccc9204cdc9613efeb2b6548d8d426bee7cfec06 
  common/src/main/java/org/apache/atlas/groovy/CastExpression.java 963724c7cefcd871918d340c25c197143d155d56 
  common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java 2d702093937b69a7d82e8d5f58fb528144377de5 
  common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java 9a726f2f82764b0a93c523467b3fa316e2dbf4e0 
  common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java 345f838635f62457b0fd42efed83fd5b96b5c16c 
  common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java 63638b73b048ed301f5a172bf7cec0e2f618d019 
  common/src/main/java/org/apache/atlas/groovy/FieldExpression.java f6d06bd194e91d332fd7418da2a9ff24342b4647 
  common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java b60edef998b4c8361b4f194539577014428483db 
  common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java 493bd3df1274fe67f47abe6606a2f447af8fa470 
  common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java 6abdbf02ed54389f6bd7d9516fa14dad5d791b8d 
  common/src/main/java/org/apache/atlas/groovy/ListExpression.java f7acaace9c4e5c9f87c596f7c11c4c41c1b13db5 
  common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 008c885686985ff38e0d1ad28a5e0f736e700a39 
  common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ee5829b39e8c4bd6c80942328100259d2762c506 
  common/src/main/java/org/apache/atlas/groovy/RangeExpression.java 7322f697474f05630398c6379ec0aa4c3ba178d6 
  common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java 75a2f8682f62d1d7bf173a6d704c7accb0c27808 
  common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java 4a6105234fe9ca33830516558a40f459ad7082f9 
  common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java 7e018f1dcffed654b958c59d70378d6178db4367 
  distro/src/conf/atlas-application.properties eea46a6abf3e8f0e1c81cd62e765d8c72e8fa623 
  pom.xml bdfd785194e35239db5feab93a5fa70ef20f84fb 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 6b6896150efe185cf4f2cb4ae005bd5aaa0b359a 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java ca1ad5df9fc01b5c50c2dcc31098beca1cdfb76d 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 3677544d73183116cd0bea4a1b04be11f19b5c1a 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/AliasFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/CallHierarchyVisitor.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandAndsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/FunctionGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinOptimization.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/HasForbiddenType.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOr.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/IsOrParent.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/OptimizationContext.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/SplitPointFinder.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/gremlin/optimizer/UpdatedExpressions.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java 1a3faf778f01582e476792780a9dbaa8a4010d29 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java 84608d9db54058ae59595acc42710b23c0c9b918 
  repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java a270b9789d67cb124e7929d495da04878f2ea35d 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala a61ff982915684b72fe7067eea2fc2ac75d31baf 
  repository/src/test/java/org/apache/atlas/repository/graph/GremlinQueryOptimizerTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/54680/diff/


Testing
-------

Ran repository and webapp tests, no regressions found.


Thanks,

Jeff Hagelberg