You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Pinal Shah <pi...@freestoneinfotech.com> on 2020/07/06 07:09:56 UTC

Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs
-----

  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 


Diff: https://reviews.apache.org/r/72646/diff/1/


Testing
-------

Manually tested
Precommit :


Thanks,

Pinal Shah


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.

> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 705 (patched)
> > <https://reviews.apache.org/r/72646/diff/2/?file=2235580#file2235580line732>
> >
> >     What is the volume of data you have tested this with?

I tested it with 5000 edge vertices.
Will vertexTotal() method be efficient than this?


- Pinal


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
-----------------------------------------------------------


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.

> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 649 (original), 662 (patched)
> > <https://reviews.apache.org/r/72646/diff/2/?file=2235580#file2235580line664>
> >
> >     Consider replacing this with AtlasGraphTraversal. This should be easy change but will give good performance mileage.
> 
> Pinal Shah wrote:
>     Thanks Ashutosh for reviewing.
>     
>     You mean to add these methods, (bothE, otherV) to AtlasGraphTraversal and make use from it?
>     How will it make difference in performance?

You are right, moving to AtlasGraphTraversal will not affect performance. It is just that it will be in 1 place.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
-----------------------------------------------------------


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.

> On July 20, 2020, 10:54 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 649 (original), 662 (patched)
> > <https://reviews.apache.org/r/72646/diff/2/?file=2235580#file2235580line664>
> >
> >     Consider replacing this with AtlasGraphTraversal. This should be easy change but will give good performance mileage.

Thanks Ashutosh for reviewing.

You mean to add these methods, (bothE, otherV) to AtlasGraphTraversal and make use from it?
How will it make difference in performance?


- Pinal


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
-----------------------------------------------------------


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221273
-----------------------------------------------------------




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 622 (patched)
<https://reviews.apache.org/r/72646/#comment310151>

    getId should be sufficient here.



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Line 649 (original), 662 (patched)
<https://reviews.apache.org/r/72646/#comment310152>

    Consider replacing this with AtlasGraphTraversal. This should be easy change but will give good performance mileage.



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 705 (patched)
<https://reviews.apache.org/r/72646/#comment310153>

    What is the volume of data you have tested this with?


- Ashutosh Mestry


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221282
-----------------------------------------------------------




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 705 (patched)
<https://reviews.apache.org/r/72646/#comment310162>

    I tested it with 5000 edge vertices.
    Will vertexTotal() method be efficient than this?


- Pinal Shah


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.

> On July 23, 2020, 10:50 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Line 607 (original), 616 (patched)
> > <https://reviews.apache.org/r/72646/diff/2/?file=2235580#file2235580line618>
> >
> >     what if out/in edges size is a lot, do you bring everything into memory just to get size? 
> >     
> >     Do you need entire edges in the list or maybe we can maintain a counter and increment?

Yes Sarath you are right but,
approximate count is count without limit/offset
I brought it to inmemory to filter out 'ACTIVE' edges incase of excludeDeleted flag
Other way is to fire another query with 'ACTIVE'filter


> On July 23, 2020, 10:50 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 682 (patched)
> > <https://reviews.apache.org/r/72646/diff/2/?file=2235580#file2235580line703>
> >
> >     Constants.GUID_PROPERTY_KEY).value() can be null (mostly not, but some bad vertices). Consider checking for null value of guid.
> >     
> >     Also, since you already have vertex, do we need additional 'endVerticesGuid' list? Can we extract 'guid' from Vertex 'v' and add it to 'resultList' is same loop. 
> >     
> >     resultList.add(entityRetriever.toAtlasEntityHeader(endVertexGuid, attributes));
> >     
> >     (or)
> >     
> >     Maybe map Vertex to AtlasVertex and directly call:
> >     
> >     resultList.add(entityRetriever.toAtlasEntityHeader(atlasVertex, attributes));

I have added  (v != null && v.property(Constants.GUID_PROPERTY_KEY).isPresent()) for the null check
Correct, we dont need extra list, will address it


- Pinal


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221343
-----------------------------------------------------------


On July 6, 2020, 9:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 9:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221343
-----------------------------------------------------------




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Line 607 (original), 616 (patched)
<https://reviews.apache.org/r/72646/#comment310237>

    what if out/in edges size is a lot, do you bring everything into memory just to get size? 
    
    Do you need entire edges in the list or maybe we can maintain a counter and increment?



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 682 (patched)
<https://reviews.apache.org/r/72646/#comment310236>

    Constants.GUID_PROPERTY_KEY).value() can be null (mostly not, but some bad vertices). Consider checking for null value of guid.
    
    Also, since you already have vertex, do we need additional 'endVerticesGuid' list? Can we extract 'guid' from Vertex 'v' and add it to 'resultList' is same loop. 
    
    resultList.add(entityRetriever.toAtlasEntityHeader(endVertexGuid, attributes));
    
    (or)
    
    Maybe map Vertex to AtlasVertex and directly call:
    
    resultList.add(entityRetriever.toAtlasEntityHeader(atlasVertex, attributes));


- Sarath Subramanian


On July 6, 2020, 2:14 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 6, 2020, 2:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221445
-----------------------------------------------------------


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1523 (patched)
<https://reviews.apache.org/r/72646/#comment310458>

    This handles edge attributes of string type. Consider handling other datatypes as well in a subsequent patch.
    
    An implementaiton similar to vertex-attribute predicate #226 - #271, which uses VertexAttributePredicateGenerator; perhaps, introducing EdgeAttributePredicateGenerator will be useful.


- Madhan Neethiraj


On July 31, 2020, 12:43 p.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 31, 2020, 12:43 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/4/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221470
-----------------------------------------------------------


Fix it, then Ship it!





intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Line 19 (original), 19 (patched)
<https://reviews.apache.org/r/72646/#comment310487>

    Only import related changes are in AtlasEntityType.java. Please review and revert.


- Madhan Neethiraj


On Aug. 4, 2020, 11:01 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated Aug. 4, 2020, 11:01 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 044179d44 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/5/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221471
-----------------------------------------------------------


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 5, 2020, 4:27 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated Aug. 5, 2020, 4:27 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 044179d44 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/6/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221472
-----------------------------------------------------------


Ship it!




Ship It!

- Nixon Rodrigues


On Aug. 5, 2020, 4:27 a.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated Aug. 5, 2020, 4:27 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
>   repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 044179d44 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/6/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

(Updated Aug. 5, 2020, 4:27 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
  repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 044179d44 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/6/

Changes: https://reviews.apache.org/r/72646/diff/5-6/


Testing (updated)
-------

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2084


Thanks,

Pinal Shah


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

(Updated Aug. 4, 2020, 11:01 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Changes
-------

added support for Edge attribute predicate


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
  repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 044179d44 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/5/

Changes: https://reviews.apache.org/r/72646/diff/4-5/


Testing
-------

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)


Thanks,

Pinal Shah


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

(Updated July 31, 2020, 12:43 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Changes
-------

Addressed review comment


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 61ee4589f 
  repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/4/

Changes: https://reviews.apache.org/r/72646/diff/3-4/


Testing
-------

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)


Thanks,

Pinal Shah


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221416
-----------------------------------------------------------




intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 539 (patched)
<https://reviews.apache.org/r/72646/#comment310381>

    I suggest to move this method to AtlasAttribute:
    
      class AtlasAttribute {
        ...
        
        public AtlasEntityType getReferencedEntityType(AtlasTypeRegistry typeRegistry) {
          return getReferencedEntityType(typeRegistry.getType(attributeDef.getTypeName()))
        }
      }


- Madhan Neethiraj


On July 29, 2020, 1:41 p.m., Pinal Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> -----------------------------------------------------------
> 
> (Updated July 29, 2020, 1:41 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
>     https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related entities**  .
> Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
> Before                    : 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/3/
> 
> 
> Testing
> -------
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

(Updated July 29, 2020, 1:41 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description (updated)
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs
-----

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/3/


Testing
-------

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)


Thanks,

Pinal Shah


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

(Updated July 29, 2020, 1:38 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Changes
-------

Addressed Review comments


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 86572598a 
  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/3/

Changes: https://reviews.apache.org/r/72646/diff/2-3/


Testing
-------

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)


Thanks,

Pinal Shah


Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
-----------------------------------------------------------

(Updated July 6, 2020, 9:14 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3876
    https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
-------

**Issue:**
Relationship api doesn't provide approximate count of the related entities in the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related entities**  .
Average time taken for the Api to search relationship entities having **5000 end vertices** with limit **500**:
Before                    : 9seconds
After applying this patch : 3seconds


Diffs
-----

  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4b9564295 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 5069d78c8 


Diff: https://reviews.apache.org/r/72646/diff/1/


Testing (updated)
-------

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 (Failed in Impala build)


Thanks,

Pinal Shah