You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by on 2019/03/12 17:48:31 UTC

Review Request 70196: ATLAS-3054: improved batch processing in notificaiton handler to avoid processing of an entity multiple times

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70196/
-----------------------------------------------------------

Review request for atlas, Ashutosh Mestry, keval bhatt, Mehul Parikh, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3054
    https://issues.apache.org/jira/browse/ATLAS-3054


Repository: atlas


Description
-------

- while batch-processing of entities in notification handler, an entity could refer to another entity processed in earlier batch. This would cause Atlas to process the already processed referenced entity to be processed again - since the reference would use internal-guid. This can be expensive for large referenced entities - such as a hive_table with 1000s of columns. This can be avoided by updating the reference to replace internal-guid with assigned-guid of the entity
- few other fixes found while processing of hive_table message


Diffs
-----

  addons/models/1000-Hadoop/1030-hive_model.json 324d71665 
  intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java 6bedf6d43 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 2557bb31b 
  intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 3ea8d805d 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 84c76d7e0 
  intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 30022173e 
  repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 9df360c89 
  repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java 51a642614 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java a62f335e7 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a6f1250fb 
  webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java d16d544d9 
  webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java ff9c9cb44 
  webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 94e099377 


Diff: https://reviews.apache.org/r/70196/diff/1/


Testing
-------

- verified that notification for a large hive_table entity, with 1000s of columns & few partition-keys, was processed correctly
- pre-commit tests run: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/971/


Thanks,

madhan

Re: Review Request 70196: ATLAS-3054: improved batch processing in notificaiton handler to avoid processing of an entity multiple times

Posted by Sarath Subramanian <sa...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70196/#review213630
-----------------------------------------------------------


Ship it!




Ship It!

- Sarath Subramanian


On March 12, 2019, 10:48 a.m., madhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70196/
> -----------------------------------------------------------
> 
> (Updated March 12, 2019, 10:48 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, keval bhatt, Mehul Parikh, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3054
>     https://issues.apache.org/jira/browse/ATLAS-3054
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> - while batch-processing of entities in notification handler, an entity could refer to another entity processed in earlier batch. This would cause Atlas to process the already processed referenced entity to be processed again - since the reference would use internal-guid. This can be expensive for large referenced entities - such as a hive_table with 1000s of columns. This can be avoided by updating the reference to replace internal-guid with assigned-guid of the entity
> - few other fixes found while processing of hive_table message
> 
> 
> Diffs
> -----
> 
>   addons/models/1000-Hadoop/1030-hive_model.json 324d71665 
>   intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java 6bedf6d43 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 2557bb31b 
>   intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 3ea8d805d 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 84c76d7e0 
>   intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 30022173e 
>   repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 9df360c89 
>   repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java 51a642614 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java a62f335e7 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a6f1250fb 
>   webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java d16d544d9 
>   webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java ff9c9cb44 
>   webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 94e099377 
> 
> 
> Diff: https://reviews.apache.org/r/70196/diff/2/
> 
> 
> Testing
> -------
> 
> - verified that notification for a large hive_table entity, with 1000s of columns & few partition-keys, was processed correctly
> - pre-commit tests run: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/971/
> 
> 
> Thanks,
> 
> madhan
> 
>