You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2017/06/01 22:28:11 UTC
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/
-----------------------------------------------------------
(Updated June 1, 2017, 10:28 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates:
- Addressed review comments.
- Implemented additional option to _AtlasImportRequest_ to conditionally import attributes.
Bugs: ATLAS-1734
https://issues.apache.org/jira/browse/ATLAS-1734
Repository: atlas
Description (updated)
-------
**Background**
Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
**Approach**
New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
_ImportService.processTypeDefs_ method now includes additional step for attribute updation.
Metrics have been updated to reflect the updates.
**CURL**
The ZIP files mentioned below can be found at: _repository/src/test/resources/_
Create a file _importOptions.json_ with contents below:
```javascript
{
"options": {
"updateAttributes": true
}
}
```
Use the following curl to perform an import with default attributes.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
```
Note the attributes of the tables.
Then perform this import with the new option.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
```
**Impact to API Consumer**
None.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d
pom.xml 4360604f
repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 27c0b5d4
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 69140e69
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java 0f711db4
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceReportingTest.java 0aaaa704
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1BulkImportPercentTest.java PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860
Diff: https://reviews.apache.org/r/58434/diff/2/
Changes: https://reviews.apache.org/r/58434/diff/1-2/
Testing
-------
**Unit tests**
- _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
- _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
- Additional unit tests: _UniqueList_.
- Import-Export APIs verified with test file set.
Thanks,
Ashutosh Mestry
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/#review178929
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java
Lines 42 (patched)
<https://reviews.apache.org/r/58434/#comment253265>
addElements() ==> updateTypes()
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java
Lines 90 (patched)
<https://reviews.apache.org/r/58434/#comment253262>
addAttributes() ==> addElements()
- Madhan Neethiraj
On June 22, 2017, 2:51 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58434/
> -----------------------------------------------------------
>
> (Updated June 22, 2017, 2:51 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1734
> https://issues.apache.org/jira/browse/ATLAS-1734
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
>
> **Approach**
> New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
> _ImportService.processTypeDefs_ method now includes additional step for attribute updation.
> Metrics have been updated to reflect the updates.
>
> **CURL**
>
> The ZIP files mentioned below can be found at: _repository/src/test/resources/_
>
>
> Create a file _importOptions.json_ with contents below:
> ```javascript
> {
> "options": {
> "updateTypeDefinition": true
> }
> }
> ```
>
> Use the following curl to perform an import with default attributes.
>
> ```javascript
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> Note the attributes of the tables.
>
> Then perform this import with the new option.
> ```javascript
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> **Impact to API Consumer**
> None.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
> repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
> repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58434/diff/4/
>
>
> Testing
> -------
>
> **Unit tests**
> - _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
> - _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
> - Additional unit tests: _UniqueList_.
> - Import-Export APIs verified with test file set.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/#review178955
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On June 27, 2017, 5:27 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58434/
> -----------------------------------------------------------
>
> (Updated June 27, 2017, 5:27 a.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1734
> https://issues.apache.org/jira/browse/ATLAS-1734
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
>
> **Approach**
> New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
> _ImportService.processTypeDefs_ method now includes additional step for attribute updation.
> Metrics have been updated to reflect the updates.
>
> **CURL**
>
> The ZIP files mentioned below can be found at: _repository/src/test/resources/_
>
>
> Create a file _importOptions.json_ with contents below:
> ```javascript
> {
> "options": {
> "updateTypeDefinition": true
> }
> }
> ```
>
> Use the following curl to perform an import with default attributes.
>
> ```javascript
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> Note the attributes of the tables.
>
> Then perform this import with the new option.
> ```javascript
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> **Impact to API Consumer**
> None.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 612c445f
> repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
> repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58434/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> - _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
> - _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
> - Additional unit tests: _UniqueList_.
> - Import-Export APIs verified with test file set.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/
-----------------------------------------------------------
(Updated June 27, 2017, 5:27 a.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates include:
- Addressed review comments.
- Fixed unit test by adding execution order.
- Unit test refactoring to reduce clutter.
Bugs: ATLAS-1734
https://issues.apache.org/jira/browse/ATLAS-1734
Repository: atlas
Description
-------
**Background**
Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
**Approach**
New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
_ImportService.processTypeDefs_ method now includes additional step for attribute updation.
Metrics have been updated to reflect the updates.
**CURL**
The ZIP files mentioned below can be found at: _repository/src/test/resources/_
Create a file _importOptions.json_ with contents below:
```javascript
{
"options": {
"updateTypeDefinition": true
}
}
```
Use the following curl to perform an import with default attributes.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
```
Note the attributes of the tables.
Then perform this import with the new option.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
```
**Impact to API Consumer**
None.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 612c445f
repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
Diff: https://reviews.apache.org/r/58434/diff/5/
Changes: https://reviews.apache.org/r/58434/diff/4-5/
Testing
-------
**Unit tests**
- _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
- _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
- Additional unit tests: _UniqueList_.
- Import-Export APIs verified with test file set.
Thanks,
Ashutosh Mestry
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/
-----------------------------------------------------------
(Updated June 22, 2017, 2:51 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Includes:
- Addressed review comments.
Bugs: ATLAS-1734
https://issues.apache.org/jira/browse/ATLAS-1734
Repository: atlas
Description
-------
**Background**
Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
**Approach**
New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
_ImportService.processTypeDefs_ method now includes additional step for attribute updation.
Metrics have been updated to reflect the updates.
**CURL**
The ZIP files mentioned below can be found at: _repository/src/test/resources/_
Create a file _importOptions.json_ with contents below:
```javascript
{
"options": {
"updateTypeDefinition": true
}
}
```
Use the following curl to perform an import with default attributes.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
```
Note the attributes of the tables.
Then perform this import with the new option.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
```
**Impact to API Consumer**
None.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
Diff: https://reviews.apache.org/r/58434/diff/4/
Changes: https://reviews.apache.org/r/58434/diff/3-4/
Testing
-------
**Unit tests**
- _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
- _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
- Additional unit tests: _UniqueList_.
- Import-Export APIs verified with test file set.
Thanks,
Ashutosh Mestry
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/#review178631
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
Lines 55 (patched)
<https://reviews.apache.org/r/58434/#comment252753>
Since 'importTypeDefProcessor' is used only from line #160 below, is it necessary to have it as a member?
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
Line 142 (original), 139 (patched)
<https://reviews.apache.org/r/58434/#comment252754>
This check is performed in importTypeDefProcessor.processTypes() as well. Consider removing from one of these places.
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
Lines 219 (patched)
<https://reviews.apache.org/r/58434/#comment252755>
Only whitespace change in this file. Please revert.
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
Lines 46 (patched)
<https://reviews.apache.org/r/58434/#comment252756>
Should the double quotes be escaped here as well - similar to previous values?
"\"true\""
- Madhan Neethiraj
On June 22, 2017, 4:27 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58434/
> -----------------------------------------------------------
>
> (Updated June 22, 2017, 4:27 a.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-1734
> https://issues.apache.org/jira/browse/ATLAS-1734
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
>
> **Approach**
> New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
> _ImportService.processTypeDefs_ method now includes additional step for attribute updation.
> Metrics have been updated to reflect the updates.
>
> **CURL**
>
> The ZIP files mentioned below can be found at: _repository/src/test/resources/_
>
>
> Create a file _importOptions.json_ with contents below:
> ```javascript
> {
> "options": {
> "updateTypeDefinition": true
> }
> }
> ```
>
> Use the following curl to perform an import with default attributes.
>
> ```javascript
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> Note the attributes of the tables.
>
> Then perform this import with the new option.
> ```javascript
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> **Impact to API Consumer**
> None.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 75e91320
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
> repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
> repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58434/diff/3/
>
>
> Testing
> -------
>
> **Unit tests**
> - _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
> - _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
> - Additional unit tests: _UniqueList_.
> - Import-Export APIs verified with test file set.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/
-----------------------------------------------------------
(Updated June 22, 2017, 4:27 a.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Includes:
- Modification to test data provided within _Description_ section.
Bugs: ATLAS-1734
https://issues.apache.org/jira/browse/ATLAS-1734
Repository: atlas
Description (updated)
-------
**Background**
Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
**Approach**
New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
_ImportService.processTypeDefs_ method now includes additional step for attribute updation.
Metrics have been updated to reflect the updates.
**CURL**
The ZIP files mentioned below can be found at: _repository/src/test/resources/_
Create a file _importOptions.json_ with contents below:
```javascript
{
"options": {
"updateTypeDefinition": true
}
}
```
Use the following curl to perform an import with default attributes.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
```
Note the attributes of the tables.
Then perform this import with the new option.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
```
**Impact to API Consumer**
None.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 75e91320
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
Diff: https://reviews.apache.org/r/58434/diff/3/
Testing
-------
**Unit tests**
- _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
- _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
- Additional unit tests: _UniqueList_.
- Import-Export APIs verified with test file set.
Thanks,
Ashutosh Mestry
Re: Review Request 58434: Import API: Improved the API to Update
Existing Types Attributes
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58434/
-----------------------------------------------------------
(Updated June 16, 2017, 5:55 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Includes:
- Addressed review comments.
- Refactored _ImportService_: Moved _typeDefinition_ processing to separate class.
- Added option to _AtlasImportRequest_ to choose typeDef import.
- Added unit tests to verify the new option.
Bugs: ATLAS-1734
https://issues.apache.org/jira/browse/ATLAS-1734
Repository: atlas
Description
-------
**Background**
Existing version of Import API allows for importing types that are not already present in the system being imported in. This causes import to fail in the cases where the data being imported happens to have the additional attribute.
**Approach**
New class _TypeAttributeDifference_ does a attribute diff betweeing 2 types.
_ImportService.processTypeDefs_ method now includes additional step for attribute updation.
Metrics have been updated to reflect the updates.
**CURL**
The ZIP files mentioned below can be found at: _repository/src/test/resources/_
Create a file _importOptions.json_ with contents below:
```javascript
{
"options": {
"updateAttributes": true
}
}
```
Use the following curl to perform an import with default attributes.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@salesNewTypeAttrs.zip "http://localhost:21000/api/atlas/admin/import"
```
Note the attributes of the tables.
Then perform this import with the new option.
```javascript
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@importOptions.json -F data=@salesNewTypeAttrs-next.zip "http://localhost:21000/api/atlas/admin/import"
```
**Impact to API Consumer**
None.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java b19f7097
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 7744c781
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 4ffbb88c
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 75e91320
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java 0844bf0d
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a6fed628
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/UniqueListTest.java PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs-next.zip PRE-CREATION
repository/src/test/resources/salesNewTypeAttrs.zip PRE-CREATION
Diff: https://reviews.apache.org/r/58434/diff/3/
Changes: https://reviews.apache.org/r/58434/diff/2-3/
Testing
-------
**Unit tests**
- _ImportServiceTest_ Updated to handle the new case. New light-weight _zip_ files added test resources to verify the functionality.
- _TypeAttributeDifferenceTest_ Unit test for _TypeAttributeDifference_ class.
- Additional unit tests: _UniqueList_.
- Import-Export APIs verified with test file set.
Thanks,
Ashutosh Mestry