You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Mandar Ambawane <ma...@freestoneinfotech.com> on 2023/03/09 11:26:40 UTC
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
-----------------------------------------------------------
(Updated March 9, 2023, 11:26 a.m.)
Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah.
Changes
-------
Code cleanup
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
-------
This ticket tracks the Server side changes required for the Search result download task.
From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
The generated files will be stored in the specific location (which is configurable).
A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/2/
Changes: https://reviews.apache.org/r/74341/diff/1-2/
Testing
-------
Thanks,
Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225375
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 76 (patched)
<https://reviews.apache.org/r/74341/#comment314016>
Please use AtlasConfiguration.SEARCH_MAX_LIMIT
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 87 (patched)
<https://reviews.apache.org/r/74341/#comment314017>
Is this check needed?
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 151 (patched)
<https://reviews.apache.org/r/74341/#comment314018>
We should use QueryParams.getNormalizedParams()
- Pinal Shah
On April 12, 2023, 6:21 a.m., Mandar Ambawane wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> -----------------------------------------------------------
>
> (Updated April 12, 2023, 6:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This ticket tracks the Server side changes required for the Search result download task.
>
> From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
>
> The generated files will be stored in the specific location (which is configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/6/
>
>
> Testing
> -------
>
> PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Jayendra Parab <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225384
-----------------------------------------------------------
Ship it!
Ship It!
- Jayendra Parab
On April 14, 2023, 8 a.m., Mandar Ambawane wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> -----------------------------------------------------------
>
> (Updated April 14, 2023, 8 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This ticket tracks the Server side changes required for the Search result download task.
>
> From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
>
> The generated files will be stored in the specific location (which is configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/8/
>
>
> Testing
> -------
>
> PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225383
-----------------------------------------------------------
Ship it!
Ship It!
- Pinal Shah
On April 14, 2023, 8 a.m., Mandar Ambawane wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> -----------------------------------------------------------
>
> (Updated April 14, 2023, 8 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This ticket tracks the Server side changes required for the Search result download task.
>
> From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
>
> The generated files will be stored in the specific location (which is configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/8/
>
>
> Testing
> -------
>
> PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Mandar Ambawane <ma...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
-----------------------------------------------------------
(Updated April 14, 2023, 8 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
Changes
-------
Addressed review comments
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
-------
This ticket tracks the Server side changes required for the Search result download task.
From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
The generated files will be stored in the specific location (which is configurable).
A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/7/
Changes: https://reviews.apache.org/r/74341/diff/6-7/
Testing
-------
PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
Request to generate csv file with default columns seen on UI:
curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
Request to generate csv file with additional columns selected from UI:
curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
Thanks,
Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
> On April 13, 2023, 10:37 a.m., Pinal Shah wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
> > Lines 166 (patched)
> > <https://reviews.apache.org/r/74341/diff/6/?file=2275775#file2275775line166>
> >
> > please explain the need of this?
is it needed for Header Names?
- Pinal
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225376
-----------------------------------------------------------
On April 12, 2023, 6:21 a.m., Mandar Ambawane wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> -----------------------------------------------------------
>
> (Updated April 12, 2023, 6:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This ticket tracks the Server side changes required for the Search result download task.
>
> From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
>
> The generated files will be stored in the specific location (which is configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/6/
>
>
> Testing
> -------
>
> PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225376
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 156 (patched)
<https://reviews.apache.org/r/74341/#comment314020>
Please format the line of code for all the changes of this patch like:
String query = (String) parameters.get(QUERY_KEY);
String typeName = (String) parameters.get(TYPE_NAME_KEY);
String classification = (String) parameters.get(CLASSIFICATION_KEY);
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 166 (patched)
<https://reviews.apache.org/r/74341/#comment314019>
please explain the need of this?
- Pinal Shah
On April 12, 2023, 6:21 a.m., Mandar Ambawane wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> -----------------------------------------------------------
>
> (Updated April 12, 2023, 6:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This ticket tracks the Server side changes required for the Search result download task.
>
> From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
>
> The generated files will be stored in the specific location (which is configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/6/
>
>
> Testing
> -------
>
> PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u <username>:<password> --request POST "<host>:<port>/api/atlas/v2/search/basic/download/create_file" --header 'Content-Type: application/json' --data-raw '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Mandar Ambawane <ma...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
-----------------------------------------------------------
(Updated April 12, 2023, 6:21 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
Changes
-------
Addressed review comments
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
-------
This ticket tracks the Server side changes required for the Search result download task.
From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
The generated files will be stored in the specific location (which is configurable).
A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/6/
Changes: https://reviews.apache.org/r/74341/diff/5-6/
Testing
-------
PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
Thanks,
Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Mandar Ambawane <ma...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
-----------------------------------------------------------
(Updated April 10, 2023, 11:07 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
Changes
-------
Addressed review comments
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
-------
This ticket tracks the Server side changes required for the Search result download task.
From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
The generated files will be stored in the specific location (which is configurable).
A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/5/
Changes: https://reviews.apache.org/r/74341/diff/4-5/
Testing (updated)
-------
PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
Thanks,
Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Jayendra Parab <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225358
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 491 (patched)
<https://reviews.apache.org/r/74341/#comment314006>
Consider having a constructor to create AtlasSearchDownloadRecord with the values set here and also in the for-loop below.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 71 (patched)
<https://reviews.apache.org/r/74341/#comment314008>
Consider prefixing all the download search related properties with atlas.download.search, so that the the properties aren't confused with any other search related properties.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 72 (patched)
<https://reviews.apache.org/r/74341/#comment314007>
Consider using sub-dir inside atlas.home directory as the default directory path.
"/tmp" path is accessible for all users.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java
Lines 55 (patched)
<https://reviews.apache.org/r/74341/#comment314010>
Consider having only the expiry time property, not the property for cleanup
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
Lines 108 (patched)
<https://reviews.apache.org/r/74341/#comment314009>
Is graph.commit() required for a read-only operation?
- Jayendra Parab
On March 30, 2023, 2:37 p.m., Mandar Ambawane wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> -----------------------------------------------------------
>
> (Updated March 30, 2023, 2:37 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This ticket tracks the Server side changes required for the Search result download task.
>
> From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
>
> The generated files will be stored in the specific location (which is configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Mandar Ambawane <ma...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
-----------------------------------------------------------
(Updated March 30, 2023, 2:37 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
Changes
-------
1. Pending task limit added.
2. Search result limit added.
3. File name changed.
4. Code cleanup.
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
-------
This ticket tracks the Server side changes required for the Search result download task.
From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
The generated files will be stored in the specific location (which is configurable).
A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 5b4bf71cc
repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/4/
Changes: https://reviews.apache.org/r/74341/diff/3-4/
Testing
-------
Thanks,
Mandar Ambawane
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
Posted by Mandar Ambawane <ma...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/
-----------------------------------------------------------
(Updated March 20, 2023, 6:37 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah, Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
Changes
-------
1. Code cleanup.
2. DSL query support.
3. CSV file name change.
Bugs: ATLAS-4733
https://issues.apache.org/jira/browse/ATLAS-4733
Repository: atlas
Description
-------
This ticket tracks the Server side changes required for the Search result download task.
From server side, each download request is considered as a separate thread which will run in the background and will generate the csv file according to the inputs provided.
The generated files will be stored in the specific location (which is configurable).
A scheduler will be run to delete the old files. In this case, the scheduler interval and file expiry time is also configurable.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java d94110004
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 582d97542
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
Diff: https://reviews.apache.org/r/74341/diff/3/
Changes: https://reviews.apache.org/r/74341/diff/2-3/
Testing
-------
Thanks,
Mandar Ambawane