You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Apoorv Naik <na...@gmail.com> on 2017/05/24 01:06:47 UTC
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review175870
-----------------------------------------------------------
This should help dealing with multi-part data
```xml
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-multipart</artifactId>
<version>1.8</version>
</dependency>
```
A sample of java REST implementation
```java
@POST
@Path("multipart")
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Produces(Servlets.JSON_MEDIA_TYPE)
public String processMultipart(@FormDataParam("meta") String jsonMeta, @FormDataParam("data") InputStream inputStream) {
return null;
}
```
intg/pom.xml
Lines 70 (patched)
<https://reviews.apache.org/r/56959/#comment249198>
Might not be needed in intg module. Please review.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java
Lines 160 (patched)
<https://reviews.apache.org/r/56959/#comment249199>
Potential NPE.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java
Lines 166 (patched)
<https://reviews.apache.org/r/56959/#comment249200>
Typo
repalce -> replace
intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java
Lines 20 (patched)
<https://reviews.apache.org/r/56959/#comment249201>
I think it's better to use the Guava Function interface or java.util.Function interface as it's exactly same.
- Apoorv Naik
On May 23, 2017, 10:11 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 23, 2017, 10:11 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
>
> CURL
> ====
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
> ```
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/pom.xml 7f3ab12b
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 91966f8f
>
>
> Diff: https://reviews.apache.org/r/56959/diff/3/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 56959: Import API: Added Support for Transforming
Entities During Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 24, 2017, 1:06 a.m., Apoorv Naik wrote:
> > This should help dealing with multi-part data
> >
> > ```xml
> > <dependency>
> > <groupId>com.sun.jersey.contribs</groupId>
> > <artifactId>jersey-multipart</artifactId>
> > <version>1.8</version>
> > </dependency>
> > ```
> >
> > A sample of java REST implementation
> > ```java
> > @POST
> > @Path("multipart")
> > @Consumes(MediaType.MULTIPART_FORM_DATA)
> > @Produces(Servlets.JSON_MEDIA_TYPE)
> > public String processMultipart(@FormDataParam("meta") String jsonMeta, @FormDataParam("data") InputStream inputStream) {
> > return null;
> > }
> > ```
Thanks a lot! This is really useful.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review175870
-----------------------------------------------------------
On May 23, 2017, 10:11 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
>
> (Updated May 23, 2017, 10:11 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
>
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case.
> * Additional classes that unit test these classes have been added.
>
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
>
> CURL
> ====
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
> ```
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
> ```
>
> Note: _-g_ needs to be added to CURL calls to prevent gobble check.
>
> Documentation
> =============
> Need to update TWIKI documents.
>
>
> Diffs
> -----
>
> intg/pom.xml 7f3ab12b
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java PRE-CREATION
> intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 91966f8f
>
>
> Diff: https://reviews.apache.org/r/56959/diff/3/
>
>
> Testing
> -------
>
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
>
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
>
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>