You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Aadarsh Jajodia <aa...@gmail.com> on 2019/04/12 00:40:00 UTC
Re: Review Request 70462: ATLAS-3133 : Adding support for Process
Executions in Atlas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70462/
-----------------------------------------------------------
(Updated April 12, 2019, 12:39 a.m.)
Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
Adding the Atlas JIRA to the review
Summary (updated)
-----------------
ATLAS-3133 : Adding support for Process Executions in Atlas
Bugs: ATLAS-3133
https://issues.apache.org/jira/browse/ATLAS-3133
Repository: atlas
Description
-------
Adding support for Process Executions in Atlas
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 31346d0954140cd8bda690dc9079e0913f7b9d7d
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java d61f1d74e3238e0a7474de67c0400c108d8919ea
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java 674a89f6e4852dc30c29c5681854ec3ba8611f35
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java 94010d0cb1a7a5c48b71f6d77c5e1a8f5cfcf013
addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 002b90839f78dc843b5aca56042c3decd299bed8
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 142e35561fff42f7672c5b5136af1173049580b3
addons/models/0000-Area0/0010-base_model.json 8b41ee89f3a7288bb4cdad3df6887242b40b68d2
addons/models/1000-Hadoop/1030-hive_model.json e96443382a587411d1207d6e1157ea65350cbdba
Diff: https://reviews.apache.org/r/70462/diff/2/
Changes: https://reviews.apache.org/r/70462/diff/1-2/
Testing
-------
We want to add support for Process Executions in Atlas. With the help of process executions attributes specific to each execution(like startTime, endTime, queryText) will be captured in the execution object. With the current model of Atlas this is lost since each execution overrides the previous one. To solve this problem. We are creating 2 new entity definitions and one relations definition. We create an entity definition called a process execution and another one called hive process execution. The hive process execution has all the attributes which are specific to each execution. We also create a relationship defintion between a hive process and a hive process execution as a many to one relation. The lineage does not get affected in this since the inputs and outputs are still attached to the hive process. Hive Process can now be thought of as a grouping of multiple executions. There is no lineage shown for a hive process execution. The criteria for grouping is based on the qualifie
dName of a Hive Process. As long as the qualified Name of a hive process remains the same, each execution gets mapped to the same hive process.
Thanks,
Aadarsh Jajodia
Re: Review Request 70462: ATLAS-3133 : Adding support for Process
Executions in Atlas
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70462/#review214611
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
Lines 616 (patched)
<https://reviews.apache.org/r/70462/#comment300783>
Instead of setting empty value (HIVE_PROCESS_REDUNDANT_FIELD_VALUE) to these attributes, it will be simpler to not set these attributes at all. i.e. remove lines #616 - #621.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
Lines 636 (patched)
<https://reviews.apache.org/r/70462/#comment300784>
Please replace ":" with QNAME_SEP_PROCESS
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
Lines 638 (patched)
<https://reviews.apache.org/r/70462/#comment300785>
value for attribute 'name' doesn't need to be appended with 'endTime'. It can simply be the query-string. Please review and update.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
Line 615 (original), 646 (patched)
<https://reviews.apache.org/r/70462/#comment300786>
Consider caching hostName, perhaps in AtlasHookContext, instead of calling InetAddress.getLocalHost().getHostName() for every process-execution.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java
Lines 46 (patched)
<https://reviews.apache.org/r/70462/#comment300787>
Please add new enums to end of the list (..a practice from good-old C/C++ days!).
addons/models/0000-Area0/0010-base_model.json
Lines 329 (patched)
<https://reviews.apache.org/r/70462/#comment300788>
To be consistent with other base types (like AtlasServer, AtlasUserProfile, AtlasUserSavedSearch, ExportImportAuditEntry), consider renaming "Process_Execution" to "ProcessExecution".
- Madhan Neethiraj
On April 12, 2019, 12:39 a.m., Aadarsh Jajodia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70462/
> -----------------------------------------------------------
>
> (Updated April 12, 2019, 12:39 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3133
> https://issues.apache.org/jira/browse/ATLAS-3133
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Adding support for Process Executions in Atlas
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 31346d0954140cd8bda690dc9079e0913f7b9d7d
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java d61f1d74e3238e0a7474de67c0400c108d8919ea
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java 674a89f6e4852dc30c29c5681854ec3ba8611f35
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java 94010d0cb1a7a5c48b71f6d77c5e1a8f5cfcf013
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 002b90839f78dc843b5aca56042c3decd299bed8
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 142e35561fff42f7672c5b5136af1173049580b3
> addons/models/0000-Area0/0010-base_model.json 8b41ee89f3a7288bb4cdad3df6887242b40b68d2
> addons/models/1000-Hadoop/1030-hive_model.json e96443382a587411d1207d6e1157ea65350cbdba
>
>
> Diff: https://reviews.apache.org/r/70462/diff/2/
>
>
> Testing
> -------
>
> We want to add support for Process Executions in Atlas. With the help of process executions attributes specific to each execution(like startTime, endTime, queryText) will be captured in the execution object. With the current model of Atlas this is lost since each execution overrides the previous one. To solve this problem. We are creating 2 new entity definitions and one relations definition. We create an entity definition called a process execution and another one called hive process execution. The hive process execution has all the attributes which are specific to each execution. We also create a relationship defintion between a hive process and a hive process execution as a many to one relation. The lineage does not get affected in this since the inputs and outputs are still attached to the hive process. Hive Process can now be thought of as a grouping of multiple executions. There is no lineage shown for a hive process execution. The criteria for grouping is based on the qualif
iedName of a Hive Process. As long as the qualified Name of a hive process remains the same, each execution gets mapped to the same hive process.
>
>
> Thanks,
>
> Aadarsh Jajodia
>
>
Re: Review Request 70462: ATLAS-3133 : Adding support for Process
Executions in Atlas
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70462/#review214615
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
Line 117 (original), 117 (patched)
<https://reviews.apache.org/r/70462/#comment300791>
for every process entity creation, we also create a process_execution entity, should we combine the method to return both entities?
something like:
List<AtlasEntity> processes = getHiveProcessEntities(inputs, outputs);
addons/models/0000-Area0/0010-base_model.json
Lines 334 (patched)
<https://reviews.apache.org/r/70462/#comment300789>
typeVersion => 1.0
addons/models/1000-Hadoop/1030-hive_model.json
Lines 646 (patched)
<https://reviews.apache.org/r/70462/#comment300790>
hive_process_process_execution => hive_process_process_executions
- Sarath Subramanian
On April 11, 2019, 5:39 p.m., Aadarsh Jajodia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70462/
> -----------------------------------------------------------
>
> (Updated April 11, 2019, 5:39 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3133
> https://issues.apache.org/jira/browse/ATLAS-3133
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Adding support for Process Executions in Atlas
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 31346d0954140cd8bda690dc9079e0913f7b9d7d
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java d61f1d74e3238e0a7474de67c0400c108d8919ea
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java 674a89f6e4852dc30c29c5681854ec3ba8611f35
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java 94010d0cb1a7a5c48b71f6d77c5e1a8f5cfcf013
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 002b90839f78dc843b5aca56042c3decd299bed8
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 142e35561fff42f7672c5b5136af1173049580b3
> addons/models/0000-Area0/0010-base_model.json 8b41ee89f3a7288bb4cdad3df6887242b40b68d2
> addons/models/1000-Hadoop/1030-hive_model.json e96443382a587411d1207d6e1157ea65350cbdba
>
>
> Diff: https://reviews.apache.org/r/70462/diff/2/
>
>
> Testing
> -------
>
> We want to add support for Process Executions in Atlas. With the help of process executions attributes specific to each execution(like startTime, endTime, queryText) will be captured in the execution object. With the current model of Atlas this is lost since each execution overrides the previous one. To solve this problem. We are creating 2 new entity definitions and one relations definition. We create an entity definition called a process execution and another one called hive process execution. The hive process execution has all the attributes which are specific to each execution. We also create a relationship defintion between a hive process and a hive process execution as a many to one relation. The lineage does not get affected in this since the inputs and outputs are still attached to the hive process. Hive Process can now be thought of as a grouping of multiple executions. There is no lineage shown for a hive process execution. The criteria for grouping is based on the qualif
iedName of a Hive Process. As long as the qualified Name of a hive process remains the same, each execution gets mapped to the same hive process.
>
>
> Thanks,
>
> Aadarsh Jajodia
>
>