You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry via Review Board <no...@reviews.apache.org> on 2021/05/14 04:57:33 UTC
Review Request 73356: Improve Tag Propagation in Entities Involved in
Several Lineages
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73356/
-----------------------------------------------------------
Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath Subramanian.
Bugs: ATLAS-4285
https://issues.apache.org/jira/browse/ATLAS-4285
Repository: atlas
Description
-------
**Steps to Duplicate**
_Setup_:
- Create a linear lineage using chained CTAS queries via Hive or REST APIs E.g.
```
create view abcd_vw_1 as select * from abc;
create view abcd_vw_2 as select * from abcd_vw_1;
create view abcd_vw_3 as select * from abcd_vw_2;
.
.
.
```
This should result in a long lineage.
- Create 3 classifications, say C1, C2, C3.
- Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3.
Expected result: abcd_vw_n should have all the 3 classifications propagated.
**Aproach**
Modify _propagateClassification_ to lock all the impacted vertices. Requests attempting to modify the same entity will thus be blocked.
Diffs
-----
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 7984a34a4
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java 8a81dc952
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java f86cbc722
repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 38f2cc9cd
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 2756504da
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java 84aefc9ef
Diff: https://reviews.apache.org/r/73356/diff/1/
Testing
-------
**Manual tests**
Used the scenario described above. See the attached file used to generate lineage.
Thanks,
Ashutosh Mestry
Re: Review Request 73356: Improve Tag Propagation in Entities
Involved in Several Lineages
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73356/#review223020
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On May 14, 2021, 9:48 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73356/
> -----------------------------------------------------------
>
> (Updated May 14, 2021, 9:48 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4285
> https://issues.apache.org/jira/browse/ATLAS-4285
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Steps to Duplicate**
> _Setup_:
> - Create a linear lineage using chained CTAS queries via Hive or REST APIs E.g.
> ```
> create view abcd_vw_1 as select * from abc;
> create view abcd_vw_2 as select * from abcd_vw_1;
> create view abcd_vw_3 as select * from abcd_vw_2;
>
> .
> .
> .
>
> ```
> This should result in a long lineage.
> - Create 3 classifications, say C1, C2, C3.
> - Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3.
>
> Expected result: abcd_vw_n should have all the 3 classifications propagated.
>
> **Aproach**
> Modify _propagateClassification_ to lock all the impacted vertices. Requests attempting to modify the same entity will thus be blocked.
>
> **Additional Information**
> - Discovered object creation sequence change for _TaskMangement_ and _TaskFactoryRegistry_. This causes pending tasks not to be scheduled.
> - Removed _ClassificationPropagationTasks.Update_ since it is not used.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 7984a34a4
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java 8a81dc952
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java f86cbc722
> repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 38f2cc9cd
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 2756504da
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java 84aefc9ef
>
>
> Diff: https://reviews.apache.org/r/73356/diff/1/
>
>
> Testing
> -------
>
> **Manual tests**
> Used the scenario described above. See the attached file used to generate lineage.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 73356: Improve Tag Propagation in Entities
Involved in Several Lineages
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73356/
-----------------------------------------------------------
(Updated May 14, 2021, 4:48 p.m.)
Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath Subramanian.
Changes
-------
Updates include: Addtional Information section.
Bugs: ATLAS-4285
https://issues.apache.org/jira/browse/ATLAS-4285
Repository: atlas
Description (updated)
-------
**Steps to Duplicate**
_Setup_:
- Create a linear lineage using chained CTAS queries via Hive or REST APIs E.g.
```
create view abcd_vw_1 as select * from abc;
create view abcd_vw_2 as select * from abcd_vw_1;
create view abcd_vw_3 as select * from abcd_vw_2;
.
.
.
```
This should result in a long lineage.
- Create 3 classifications, say C1, C2, C3.
- Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3.
Expected result: abcd_vw_n should have all the 3 classifications propagated.
**Aproach**
Modify _propagateClassification_ to lock all the impacted vertices. Requests attempting to modify the same entity will thus be blocked.
**Additional Information**
- Discovered object creation sequence change for _TaskMangement_ and _TaskFactoryRegistry_. This causes pending tasks not to be scheduled.
- Removed _ClassificationPropagationTasks.Update_ since it is not used.
Diffs
-----
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 7984a34a4
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java 8a81dc952
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java f86cbc722
repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 38f2cc9cd
repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 2756504da
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java 84aefc9ef
Diff: https://reviews.apache.org/r/73356/diff/1/
Testing
-------
**Manual tests**
Used the scenario described above. See the attached file used to generate lineage.
Thanks,
Ashutosh Mestry