You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2017/05/23 22:03:08 UTC
Re: Review Request 56959: Import Transform (Draft)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 23, 2017, 10:03 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Added link to JIRA.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description (updated)
-------
Please refer to [ATLAS-1825|https://issues.apache.org/jira/browse/ATLAS-1825] for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
CURL
====
```
curl -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
```
```
curl -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
```
Diffs
-----
intg/pom.xml 7f3ab12b
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 91966f8f
Diff: https://reviews.apache.org/r/56959/diff/2/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176039
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 45 (patched)
<https://reviews.apache.org/r/56959/#comment249370>
Is this used anymore ?
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 83 (patched)
<https://reviews.apache.org/r/56959/#comment249371>
Same here
- Apoorv Naik
On May 24, 2017, 11:15 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 24, 2017, 11:15 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
> pom.xml a93b8ad9
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/5/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176041
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Line 50 (original), 55 (patched)
<https://reviews.apache.org/r/56959/#comment249372>
AtlasBaseException seems to be not thrown. Please review and remove it if not thrown.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 92 (patched)
<https://reviews.apache.org/r/56959/#comment249373>
Given an instance of AtlasImportRequest will be serialized/deserialized during REST calls, values in 'options' map will not be typed - as this line expects the value of type AtlasImportTransforms. Please review and update to handle this.
I would suggest to keep the classes used in REST APIs to only have simple get/set. Adding any further methods/expectations might be troublesome.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 100 (patched)
<https://reviews.apache.org/r/56959/#comment249374>
Consider removing this method; instead use AtlasType.fromJson().
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java
Lines 45 (patched)
<https://reviews.apache.org/r/56959/#comment249375>
The enum implementation looks too complex to pass around instances via REST API. Consider keeping the classes/enums passed via REST simple i.e. strictly data-only. Any functionality/constraits should be handled in the application.
- Madhan Neethiraj
On May 25, 2017, 5:16 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 25, 2017, 5:16 a.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
> pom.xml a93b8ad9
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/6/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 30, 2017, 6:56 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java
> > Lines 29 (patched)
> > <https://reviews.apache.org/r/56959/diff/7/?file=1732905#file1732905line29>
> >
> > possible NPE, add null checks for entityWithExtInfo
Added test.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176328
-----------------------------------------------------------
On May 25, 2017, 9:32 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 25, 2017, 9:32 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
> "transforms": {
> "hive_db": {
> "qualifiedName": [
> "replace:@cl1:@cl2"
> ]
> },
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> pom.xml a93b8ad9
> repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/7/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176328
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java
Lines 29 (patched)
<https://reviews.apache.org/r/56959/#comment249730>
possible NPE, add null checks for entityWithExtInfo
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java
Lines 67 (patched)
<https://reviews.apache.org/r/56959/#comment249725>
this line could be part of a new else block above. It would be good to pass the "key" string in the error message to know the invalid transformer name in log.
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
Line 31 (original), 31 (patched)
<https://reviews.apache.org/r/56959/#comment249726>
revert these changes, it is already committed.
- Sarath Subramanian
On May 25, 2017, 2:32 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 25, 2017, 2:32 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
> "transforms": {
> "hive_db": {
> "qualifiedName": [
> "replace:@cl1:@cl2"
> ]
> },
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> pom.xml a93b8ad9
> repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/7/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On June 2, 2017, 8:15 a.m., David Radley wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java
> > Lines 59 (patched)
> > <https://reviews.apache.org/r/56959/diff/8/?file=1734855#file1734855line59>
> >
> > How do these transformations show in lineage?
Right now they don't.
> On June 2, 2017, 8:15 a.m., David Radley wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java
> > Lines 62 (patched)
> > <https://reviews.apache.org/r/56959/diff/8/?file=1734855#file1734855line62>
> >
> > I think that uppercase class names are not usual - wht have you not just camel cased?
My rationale was that they are classes that don't have much logic. More like Enums. I will change them to be consistent.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176741
-----------------------------------------------------------
On May 30, 2017, 7:50 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 30, 2017, 7:50 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
> "transforms": {
> "hive_db": {
> "qualifiedName": [
> "replace:@cl1:@cl2"
> ]
> },
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> pom.xml 4360604f
> repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/8/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by David Radley <da...@uk.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176741
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
Lines 65 (patched)
<https://reviews.apache.org/r/56959/#comment250194>
formatting
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java
Lines 59 (patched)
<https://reviews.apache.org/r/56959/#comment250191>
How do these transformations show in lineage?
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java
Lines 62 (patched)
<https://reviews.apache.org/r/56959/#comment250192>
I think that uppercase class names are not usual - wht have you not just camel cased?
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
Lines 70 (patched)
<https://reviews.apache.org/r/56959/#comment250193>
formatting
- David Radley
On May 30, 2017, 7:50 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 30, 2017, 7:50 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
> "transforms": {
> "hive_db": {
> "qualifiedName": [
> "replace:@cl1:@cl2"
> ]
> },
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> pom.xml 4360604f
> repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/8/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176657
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 56 (patched)
<https://reviews.apache.org/r/56959/#comment250071>
Given 'options' is used in REST interface, it is better to avoid use of "Object". Instead consider using "Map<String, String>" as type for 'options'. The value, if it is a complex structure, should be a stringified-JSON - as shown below.
"options": {
"transforms": "{ \"hive_db\": { \"qualifiedName\": \"[ \"replace:@cl1:@cl2\" ]" }, \"hive_table\": { \"qualifiedName\": \"[ \"lowercase\", \"replace:@cl1:@cl2\" ]\" } }"
}
or
"options": {
"transforms:hive_db.qualifiedName": "[ \"replace:@cl1:@cl2\" ]",
"transforms:hive_table.qualifiedName": "[ \"lowercase\", \"replace:@cl1:@cl2\" ]"
}
- Madhan Neethiraj
On May 30, 2017, 7:50 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 30, 2017, 7:50 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
> "transforms": {
> "hive_db": {
> "qualifiedName": [
> "replace:@cl1:@cl2"
> ]
> },
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> pom.xml 4360604f
> repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/8/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176805
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On June 2, 2017, 4:30 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated June 2, 2017, 4:30 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
> "transforms": {
> "hive_db": {
> "qualifiedName": [
> "replace:@cl1:@cl2"
> ]
> },
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> pom.xml 4360604f
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceReportingTest.java 0aaaa704
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerJSONTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/9/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated June 2, 2017, 4:30 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates include:
- Addressed review comments.
- Modified JSON format for transforms.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{ "options": {
"transforms": {
"hive_db": {
"qualifiedName": [
"replace:@cl1:@cl2"
]
},
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
pom.xml 4360604f
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceReportingTest.java 0aaaa704
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerJSONTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/56959/diff/9/
Changes: https://reviews.apache.org/r/56959/diff/8-9/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 30, 2017, 7:50 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates:
- Addressed review comments.
- Renamed unit test suite to reflect functionality.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{ "options": {
"transforms": {
"hive_db": {
"qualifiedName": [
"replace:@cl1:@cl2"
]
},
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
pom.xml 4360604f
repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/56959/diff/8/
Changes: https://reviews.apache.org/r/56959/diff/7-8/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 25, 2017, 9:32 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates:
- Refactored _AtlasImportRequest_ to make it simpler.
- Moved class to _repository_ from _intg_.
- Addressed review comments.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description (updated)
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{ "options": {
"transforms": {
"hive_db": {
"qualifiedName": [
"replace:@cl1:@cl2"
]
},
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
pom.xml a93b8ad9
repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/56959/diff/7/
Changes: https://reviews.apache.org/r/56959/diff/6-7/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 25, 2017, 2:19 p.m., Apoorv Naik wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Line 366 (original), 367 (patched)
> > <https://reviews.apache.org/r/56959/diff/6/?file=1732372#file1732372line368>
> >
> > Can you check if the data can directly be serialized the Java POJO here ? I did see a few similar usecases on StackOverflow and they were de-serializing into the POJO directly at the interface level.
I got an exception when i tried it earlier. I have simplified the _AtlasImportRequest_ class further. Let me re-attempt it.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176074
-----------------------------------------------------------
On May 25, 2017, 5:16 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 25, 2017, 5:16 a.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
> pom.xml a93b8ad9
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/6/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176074
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 366 (original), 367 (patched)
<https://reviews.apache.org/r/56959/#comment249405>
Can you check if the data can directly be serialized the Java POJO here ? I did see a few similar usecases on StackOverflow and they were de-serializing into the POJO directly at the interface level.
- Apoorv Naik
On May 25, 2017, 5:16 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 25, 2017, 5:16 a.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.
>
> CURL
> ====
>
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
>
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
>
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
> pom.xml a93b8ad9
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
> webapp/pom.xml 4132912d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
>
>
> Diff: https://reviews.apache.org/r/56959/diff/6/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 25, 2017, 5:16 a.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Fixed problem with enum used as factory.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
"options": {},
"importTransform": {
"transforms": {
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
pom.xml a93b8ad9
repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/56959/diff/6/
Changes: https://reviews.apache.org/r/56959/diff/5-6/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 24, 2017, 11:15 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Additional text.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description (updated)
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
"options": {},
"importTransform": {
"transforms": {
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
Documentation
=============
Need to update TWIKI documents.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
pom.xml a93b8ad9
repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/56959/diff/4/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 24, 2017, 11:06 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Refactoring:
* To support muti-part input to Import API.
* Overall to reduce code.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description (updated)
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
"options": {},
"importTransform": {
"transforms": {
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION
pom.xml a93b8ad9
repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/56959/diff/4/
Changes: https://reviews.apache.org/r/56959/diff/3-4/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 24, 2017, 1:06 a.m., Apoorv Naik wrote:
> > This should help dealing with multi-part data
> >
> > ```xml
> > <dependency>
> > <groupId>com.sun.jersey.contribs</groupId>
> > <artifactId>jersey-multipart</artifactId>
> > <version>1.8</version>
> > </dependency>
> > ```
> >
> > A sample of java REST implementation
> > ```java
> > @POST
> > @Path("multipart")
> > @Consumes(MediaType.MULTIPART_FORM_DATA)
> > @Produces(Servlets.JSON_MEDIA_TYPE)
> > public String processMultipart(@FormDataParam("meta") String jsonMeta, @FormDataParam("data") InputStream inputStream) {
> > return null;
> > }
> > ```
Thanks a lot! This is really useful.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review175870
-----------------------------------------------------------
On May 23, 2017, 10:11 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 23, 2017, 10:11 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
>
> CURL
> ====
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
> ```
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/pom.xml 7f3ab12b
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 91966f8f
>
>
> Diff: https://reviews.apache.org/r/56959/diff/3/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review175870
-----------------------------------------------------------
This should help dealing with multi-part data
```xml
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-multipart</artifactId>
<version>1.8</version>
</dependency>
```
A sample of java REST implementation
```java
@POST
@Path("multipart")
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Produces(Servlets.JSON_MEDIA_TYPE)
public String processMultipart(@FormDataParam("meta") String jsonMeta, @FormDataParam("data") InputStream inputStream) {
return null;
}
```
intg/pom.xml
Lines 70 (patched)
<https://reviews.apache.org/r/56959/#comment249198>
Might not be needed in intg module. Please review.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java
Lines 160 (patched)
<https://reviews.apache.org/r/56959/#comment249199>
Potential NPE.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java
Lines 166 (patched)
<https://reviews.apache.org/r/56959/#comment249200>
Typo
repalce -> replace
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java
Lines 20 (patched)
<https://reviews.apache.org/r/56959/#comment249201>
I think it's better to use the Guava Function interface or java.util.Function interface as it's exactly same.
- Apoorv Naik
On May 23, 2017, 10:11 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 23, 2017, 10:11 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
>
> CURL
> ====
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
> ```
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/pom.xml 7f3ab12b
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 91966f8f
>
>
> Diff: https://reviews.apache.org/r/56959/diff/3/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import Transform (Draft)
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated May 23, 2017, 10:11 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Added comments to patch.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
CURL
====
```
curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
```
```
curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/pom.xml 7f3ab12b
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java PRE-CREATION
intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 91966f8f
Diff: https://reviews.apache.org/r/56959/diff/3/
Changes: https://reviews.apache.org/r/56959/diff/2-3/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry