You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Apoorv Naik <na...@gmail.com> on 2018/04/17 17:13:28 UTC
Review Request 66653: ATLAS-2578: Update metric queries for faster
execution
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66653/
-----------------------------------------------------------
Review request for atlas.
Bugs: ATLAS-2578
https://issues.apache.org/jira/browse/ATLAS-2578
Repository: atlas
Description
-------
Updated the business logic around metrics collection to allow faster collection of metrics
Diffs
-----
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java cfccc96c3
repository/src/main/java/org/apache/atlas/services/MetricsService.java 0fa68b257
repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 1eb7323ce
repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 72b7261dd
repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java cca80b5ff
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java bdc0f0327
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java b648bc1e6
Diff: https://reviews.apache.org/r/66653/diff/1/
Testing
-------
PreCommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/278/
Thanks,
Apoorv Naik
Re: Review Request 66653: ATLAS-2578: Update metric queries for faster
execution
Posted by Apoorv Naik <na...@gmail.com>.
> On April 18, 2018, 5 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java
> > Line 26 (original), 26 (patched)
> > <https://reviews.apache.org/r/66653/diff/1/?file=2004975#file2004975line26>
> >
> > Would the following be efficient, which avoids having to pass a large list of types (and T.in check on each vertex)?
> > g.V().has('__typeName').has('__state', 'ACTIVE').values('__typeName').groupCount()
> >
> > Similarly for line #29 below:
> > g.V().has('__typeName').has('__state', 'DELETED').values('__typeName').groupCount()
This query issues a warning that a full graph scan, hence the within clause
- Apoorv
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66653/#review201386
-----------------------------------------------------------
On April 18, 2018, 6:07 a.m., Apoorv Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66653/
> -----------------------------------------------------------
>
> (Updated April 18, 2018, 6:07 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-2578
> https://issues.apache.org/jira/browse/ATLAS-2578
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Updated the business logic around metrics collection to allow faster collection of metrics
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java cfccc96c3
> repository/src/main/java/org/apache/atlas/services/MetricsService.java 0fa68b257
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 1eb7323ce
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 72b7261dd
> repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java cca80b5ff
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java bdc0f0327
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java b648bc1e6
>
>
> Diff: https://reviews.apache.org/r/66653/diff/3/
>
>
> Testing
> -------
>
> PreCommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/278/
>
>
> Thanks,
>
> Apoorv Naik
>
>
Re: Review Request 66653: ATLAS-2578: Update metric queries for faster
execution
Posted by Madhan Neethiraj <ma...@apache.org>.
> On April 18, 2018, 5 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java
> > Line 26 (original), 26 (patched)
> > <https://reviews.apache.org/r/66653/diff/1/?file=2004975#file2004975line26>
> >
> > Would the following be efficient, which avoids having to pass a large list of types (and T.in check on each vertex)?
> > g.V().has('__typeName').has('__state', 'ACTIVE').values('__typeName').groupCount()
> >
> > Similarly for line #29 below:
> > g.V().has('__typeName').has('__state', 'DELETED').values('__typeName').groupCount()
>
> Apoorv Naik wrote:
> This query issues a warning that a full graph scan, hence the within clause
ok. I think current query also will endup scanning almost all vertices - given the list includes all types defined in Atlas. It will be good to check the performance of both versions. If the performance is better, the change can be committed in a subsequent patch.
Thanks!
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66653/#review201386
-----------------------------------------------------------
On April 18, 2018, 6:07 a.m., Apoorv Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66653/
> -----------------------------------------------------------
>
> (Updated April 18, 2018, 6:07 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-2578
> https://issues.apache.org/jira/browse/ATLAS-2578
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Updated the business logic around metrics collection to allow faster collection of metrics
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java cfccc96c3
> repository/src/main/java/org/apache/atlas/services/MetricsService.java 0fa68b257
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 1eb7323ce
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 72b7261dd
> repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java cca80b5ff
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java bdc0f0327
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java b648bc1e6
>
>
> Diff: https://reviews.apache.org/r/66653/diff/3/
>
>
> Testing
> -------
>
> PreCommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/278/
>
>
> Thanks,
>
> Apoorv Naik
>
>
Re: Review Request 66653: ATLAS-2578: Update metric queries for faster
execution
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66653/#review201386
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java
Line 80 (original), 76 (patched)
<https://reviews.apache.org/r/66653/#comment282608>
Consider reusing getMetric(groupKey, key), to avoid duplicate code:
public Number getNumericMetric(String groupKey, String key) {
Object obj = getMetric(groupKey, key);
return obj instanceof Number ? (Number) obj : null;
}
repository/src/main/java/org/apache/atlas/services/MetricsService.java
Lines 139 (patched)
<https://reviews.apache.org/r/66653/#comment282609>
given activeCount is Map<String, Number>, consider declaring "count" as Number.
Same for line #150 and #164 as well.
repository/src/main/java/org/apache/atlas/services/MetricsService.java
Line 106 (original), 144 (patched)
<https://reviews.apache.org/r/66653/#comment282610>
this may not be correct, as the type can be present in 'deletedCount'
repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java
Line 26 (original), 26 (patched)
<https://reviews.apache.org/r/66653/#comment282612>
Would the following be efficient, which avoids having to pass a large list of types (and T.in check on each vertex)?
g.V().has('__typeName').has('__state', 'ACTIVE').values('__typeName').groupCount()
Similarly for line #29 below:
g.V().has('__typeName').has('__state', 'DELETED').values('__typeName').groupCount()
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 107 (original), 107 (patched)
<https://reviews.apache.org/r/66653/#comment282613>
line break missing
- Madhan Neethiraj
On April 17, 2018, 5:13 p.m., Apoorv Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66653/
> -----------------------------------------------------------
>
> (Updated April 17, 2018, 5:13 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-2578
> https://issues.apache.org/jira/browse/ATLAS-2578
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Updated the business logic around metrics collection to allow faster collection of metrics
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java cfccc96c3
> repository/src/main/java/org/apache/atlas/services/MetricsService.java 0fa68b257
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 1eb7323ce
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 72b7261dd
> repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java cca80b5ff
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java bdc0f0327
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java b648bc1e6
>
>
> Diff: https://reviews.apache.org/r/66653/diff/1/
>
>
> Testing
> -------
>
> PreCommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/278/
>
>
> Thanks,
>
> Apoorv Naik
>
>
Re: Review Request 66653: ATLAS-2578: Update metric queries for faster
execution
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66653/#review201396
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On April 18, 2018, 6:07 a.m., Apoorv Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66653/
> -----------------------------------------------------------
>
> (Updated April 18, 2018, 6:07 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-2578
> https://issues.apache.org/jira/browse/ATLAS-2578
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Updated the business logic around metrics collection to allow faster collection of metrics
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java cfccc96c3
> repository/src/main/java/org/apache/atlas/services/MetricsService.java 0fa68b257
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 1eb7323ce
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 72b7261dd
> repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java cca80b5ff
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java bdc0f0327
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java b648bc1e6
>
>
> Diff: https://reviews.apache.org/r/66653/diff/3/
>
>
> Testing
> -------
>
> PreCommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/278/
>
>
> Thanks,
>
> Apoorv Naik
>
>
Re: Review Request 66653: ATLAS-2578: Update metric queries for faster
execution
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66653/
-----------------------------------------------------------
(Updated April 18, 2018, 6:07 a.m.)
Review request for atlas.
Changes
-------
Addressed review comments
Bugs: ATLAS-2578
https://issues.apache.org/jira/browse/ATLAS-2578
Repository: atlas
Description
-------
Updated the business logic around metrics collection to allow faster collection of metrics
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java cfccc96c3
repository/src/main/java/org/apache/atlas/services/MetricsService.java 0fa68b257
repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 1eb7323ce
repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 72b7261dd
repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java cca80b5ff
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java bdc0f0327
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java b648bc1e6
Diff: https://reviews.apache.org/r/66653/diff/2/
Changes: https://reviews.apache.org/r/66653/diff/1-2/
Testing
-------
PreCommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/278/
Thanks,
Apoorv Naik