You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Pinal Shah <pi...@freestoneinfotech.com> on 2021/05/07 12:40:12 UTC
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize
pagination
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73292/
-----------------------------------------------------------
(Updated May 7, 2021, 12:40 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
refactored
Bugs: ATLAS-4254
https://issues.apache.org/jira/browse/ATLAS-4254
Repository: atlas
Description (updated)
-------
**Current Working of Pagination**
- When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0
- Query is then executed from 0 offset and limit
- Filter out the results obtained from query
- Generate result set by skipping the results from 0 to the given offset
**New Working Structure**
- New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value.
- The marker should be set as '*' for the start of the page
- The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request
- if marker is set then offset should not be greater than 0
**Perfomance details**
Took usecase of ranger tagsync:
- On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s
- On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s
(both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers)
- Around 51% improvement
- Improvement is seen as the page size increases.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2
repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2
repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d
Diff: https://reviews.apache.org/r/73292/diff/3/
Changes: https://reviews.apache.org/r/73292/diff/2-3/
Testing
-------
MAnual testing
unit testcases
Thanks,
Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize
pagination
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73292/#review222954
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 466 (patched)
<https://reviews.apache.org/r/73292/#comment312123>
getNextMarker is computed 2 times. Consider local variable.
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
Lines 316 (patched)
<https://reviews.apache.org/r/73292/#comment312121>
Use camel case.
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
Lines 870 (patched)
<https://reviews.apache.org/r/73292/#comment312122>
This could be refactored into asertSearchProcessorWithMarker and asertSearchProcessorWithoutMarker.
- Ashutosh Mestry
On May 7, 2021, 12:40 p.m., Pinal Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73292/
> -----------------------------------------------------------
>
> (Updated May 7, 2021, 12:40 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4254
> https://issues.apache.org/jira/browse/ATLAS-4254
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Current Working of Pagination**
>
> - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0
> - Query is then executed from 0 offset and limit
> - Filter out the results obtained from query
> - Generate result set by skipping the results from 0 to the given offset
>
> **New Working Structure**
>
> - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value.
> - The marker should be set as '*' for the start of the page
> - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request
> - if marker is set then offset should not be greater than 0
>
>
> **Perfomance details**
>
> Took usecase of ranger tagsync:
> - On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s
> - On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s
> (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers)
> - Around 51% improvement
>
> - Improvement is seen as the page size increases.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6
> repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2
> repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d
>
>
> Diff: https://reviews.apache.org/r/73292/diff/3/
>
>
> Testing
> -------
>
> MAnual testing
> unit testcases
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize
pagination
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73292/#review223040
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On May 13, 2021, 8:43 a.m., Pinal Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73292/
> -----------------------------------------------------------
>
> (Updated May 13, 2021, 8:43 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4254
> https://issues.apache.org/jira/browse/ATLAS-4254
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Current Working of Pagination**
>
> - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0
> - Query is then executed from 0 offset and limit
> - Filter out the results obtained from query
> - Generate result set by skipping the results from 0 to the given offset
>
> **New Working Structure**
>
> - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value.
> - The marker should be set as '*' for the start of the page
> - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request
> - if marker is set then offset should not be greater than 0
>
>
> **Perfomance details**
>
> Took usecase of ranger tagsync:
> - On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s
> - On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s
> (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers)
> - Removed 'atlas.graph.index.search.max-result-set-size=150' property from atlas-application.properties
>
> - Around 25% improvement
>
> - Improvement is seen as the page size increases.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16
> repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6
> repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2
> repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2
> repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d
>
>
> Diff: https://reviews.apache.org/r/73292/diff/5/
>
>
> Testing
> -------
>
> MAnual testing
> unit testcases
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize
pagination
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73292/
-----------------------------------------------------------
(Updated May 13, 2021, 3:43 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-4254
https://issues.apache.org/jira/browse/ATLAS-4254
Repository: atlas
Description (updated)
-------
**Current Working of Pagination**
- When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0
- Query is then executed from 0 offset and limit
- Filter out the results obtained from query
- Generate result set by skipping the results from 0 to the given offset
**New Working Structure**
- New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value.
- The marker should be set as '*' for the start of the page
- The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request
- if marker is set then offset should not be greater than 0
**Perfomance details**
Took usecase of ranger tagsync:
- On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s
- On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s
(both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers)
- Removed 'atlas.graph.index.search.max-result-set-size=150' property from atlas-application.properties
- Around 25% improvement
- Improvement is seen as the page size increases.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2
repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2
repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d
Diff: https://reviews.apache.org/r/73292/diff/5/
Testing
-------
MAnual testing
unit testcases
Thanks,
Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize
pagination
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73292/
-----------------------------------------------------------
(Updated May 10, 2021, 4:56 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-4254
https://issues.apache.org/jira/browse/ATLAS-4254
Repository: atlas
Description (updated)
-------
**Current Working of Pagination**
- When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0
- Query is then executed from 0 offset and limit
- Filter out the results obtained from query
- Generate result set by skipping the results from 0 to the given offset
**New Working Structure**
- New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value.
- The marker should be set as '*' for the start of the page
- The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request
- if marker is set then offset should not be greater than 0
**Perfomance details**
Took usecase of ranger tagsync:
- On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s
- On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s
(both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers)
- Removed 'atlas.graph.index.search.max-result-set-size=150' property from atlas-application.properties
- Around 51% improvement
- Improvement is seen as the page size increases.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2
repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2
repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d
Diff: https://reviews.apache.org/r/73292/diff/4/
Testing
-------
MAnual testing
unit testcases
Thanks,
Pinal Shah
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize
pagination
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73292/
-----------------------------------------------------------
(Updated May 10, 2021, 4:43 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
addressed review comments
Bugs: ATLAS-4254
https://issues.apache.org/jira/browse/ATLAS-4254
Repository: atlas
Description
-------
**Current Working of Pagination**
- When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0
- Query is then executed from 0 offset and limit
- Filter out the results obtained from query
- Generate result set by skipping the results from 0 to the given offset
**New Working Structure**
- New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value.
- The marker should be set as '*' for the start of the page
- The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request
- if marker is set then offset should not be greater than 0
**Perfomance details**
Took usecase of ranger tagsync:
- On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s
- On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s
(both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers)
- Around 51% improvement
- Improvement is seen as the page size increases.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16
repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2
repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2
repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d
Diff: https://reviews.apache.org/r/73292/diff/4/
Changes: https://reviews.apache.org/r/73292/diff/3-4/
Testing
-------
MAnual testing
unit testcases
Thanks,
Pinal Shah