Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

[Madhan Neethiraj <ma...@apache.org>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176657
-----------------------------------------------------------




intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 56 (patched)
<https://reviews.apache.org/r/56959/#comment250071>

    Given 'options' is used in REST interface, it is better to avoid use of "Object". Instead consider using "Map<String, String>" as type for 'options'. The value, if it is a complex structure, should be a stringified-JSON - as shown below.
    
      
    "options": {
      "transforms": "{ \"hive_db\": { \"qualifiedName\": \"[ \"replace:@cl1:@cl2\" ]" }, \"hive_table\": { \"qualifiedName\": \"[ \"lowercase\", \"replace:@cl1:@cl2\" ]\" } }"
     }
    
    or
    
    "options": {
      "transforms:hive_db.qualifiedName": "[ \"replace:@cl1:@cl2\" ]",
      "transforms:hive_table.qualifiedName": "[ \"lowercase\", \"replace:@cl1:@cl2\" ]"
     }


- Madhan Neethiraj


On May 30, 2017, 7:50 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> -----------------------------------------------------------
> 
> (Updated May 30, 2017, 7:50 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1825
>     https://issues.apache.org/jira/browse/ATLAS-1825
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.
> 
> Impact
> ======
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
>     * _replace_ Implements find and replace on attribute values.
>     * _lowercase_ Converts attribute value to lower case. 
> * Additional classes that unit test these classes have been added.
> 
> Implementation Approach
> =======================
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
> 
> CURL
> ====
> 
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
>         "transforms": {
>             "hive_db": {
>                 "qualifiedName": [
>                     "replace:@cl1:@cl2"
>                 ]
>             },
>             "hive_table": {
>                 "qualifiedName": [
>                     "lowercase",
>                     "replace:@cl1:@cl2"
>                 ]
>             }
>         }
>     }
> }
> ```
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> 
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.
> 
> Notice the output from the API, it has the _AtlasRequest_ that was passed as input.
> 
> Documentation
> =============
> Need to update TWIKI documents.
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d 
>   pom.xml 4360604f 
>   repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb 
>   repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a 
>   webapp/pom.xml 4132912d 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860 
> 
> 
> Diff: https://reviews.apache.org/r/56959/diff/8/
> 
> 
> Testing
> -------
> 
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
> 
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
> 
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>