You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Sidharth Mishra <si...@gmail.com> on 2021/08/25 06:21:36 UTC
Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4400
https://issues.apache.org/jira/browse/ATLAS-4400
Repository: atlas
Description
-------
ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
Diffs
-----
common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java PRE-CREATION
Diff: https://reviews.apache.org/r/73544/diff/1/
Testing
-------
Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/817/
Thanks,
Sidharth Mishra
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/
-----------------------------------------------------------
(Updated Aug. 27, 2021, 5:20 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4400
https://issues.apache.org/jira/browse/ATLAS-4400
Repository: atlas
Description
-------
ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
Diffs
-----
common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
webapp/src/main/java/org/apache/atlas/notification/preprocessor/AWSS3V2Preprocessor.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
webapp/src/test/java/org/apache/atlas/notification/preprocessor/AWSS3V2PreprocessorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/73544/diff/3/
Testing (updated)
-------
Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/829/
Thanks,
Sidharth Mishra
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/#review223414
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On Aug. 26, 2021, 4:39 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73544/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2021, 4:39 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4400
> https://issues.apache.org/jira/browse/ATLAS-4400
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
> common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/AWSS3V2Preprocessor.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
> webapp/src/test/java/org/apache/atlas/notification/preprocessor/AWSS3V2PreprocessorTest.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/73544/diff/3/
>
>
> Testing
> -------
>
> Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/824/
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/
-----------------------------------------------------------
(Updated Aug. 26, 2021, 11:39 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4400
https://issues.apache.org/jira/browse/ATLAS-4400
Repository: atlas
Description
-------
ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
webapp/src/main/java/org/apache/atlas/notification/preprocessor/AWSS3V2Preprocessor.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
webapp/src/test/java/org/apache/atlas/notification/preprocessor/AWSS3V2PreprocessorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/73544/diff/3/
Changes: https://reviews.apache.org/r/73544/diff/2-3/
Testing
-------
Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/824/
Thanks,
Sidharth Mishra
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/#review223404
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On Aug. 25, 2021, 8:41 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73544/
> -----------------------------------------------------------
>
> (Updated Aug. 25, 2021, 8:41 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4400
> https://issues.apache.org/jira/browse/ATLAS-4400
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
> common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/AWSS3V2Preprocessor.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
>
>
> Diff: https://reviews.apache.org/r/73544/diff/2/
>
>
> Testing
> -------
>
> Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/824/
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/
-----------------------------------------------------------
(Updated Aug. 26, 2021, 3:41 a.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4400
https://issues.apache.org/jira/browse/ATLAS-4400
Repository: atlas
Description
-------
ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
webapp/src/main/java/org/apache/atlas/notification/preprocessor/AWSS3V2Preprocessor.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
Diff: https://reviews.apache.org/r/73544/diff/2/
Changes: https://reviews.apache.org/r/73544/diff/1-2/
Testing
-------
Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/824/
Thanks,
Sidharth Mishra
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/
-----------------------------------------------------------
(Updated Aug. 25, 2021, 11:39 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4400
https://issues.apache.org/jira/browse/ATLAS-4400
Repository: atlas
Description
-------
ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
Diffs
-----
common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java PRE-CREATION
Diff: https://reviews.apache.org/r/73544/diff/1/
Testing (updated)
-------
Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/824/
Thanks,
Sidharth Mishra
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sidharth Mishra <si...@gmail.com>.
> On Aug. 25, 2021, 10:18 p.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java
> > Lines 66 (patched)
> > <https://reviews.apache.org/r/73544/diff/1/?file=2252253#file2252253line66>
> >
> > consider also including the 'name' attribute from aws_s3_v2_directory type in this check:
> >
> > && objectPrefix.contains(name)
> >
> > qualifiedName : s3a://my_bucket_1/datalake/warehouse/tablespace/external/hive/employee.csv/@cm
> > objectPrefix : /datalake/warehouse/tablespace/external/hive/employee.csv/
> > name : employee.csv
We can't add name. Consider one example where location is:
qualifiedName : s3a://my_bucket_1/temp/temp/temp@cm
objectPrefix : /temp/temp/temp/
name : temp
Even current check was not enough for this. So I added check for schema + bucket + objectPrefix + "@"
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/#review223398
-----------------------------------------------------------
On Aug. 25, 2021, 11:39 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73544/
> -----------------------------------------------------------
>
> (Updated Aug. 25, 2021, 11:39 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4400
> https://issues.apache.org/jira/browse/ATLAS-4400
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
> common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/73544/diff/1/
>
>
> Testing
> -------
>
> Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/824/
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73544: ATLAS-4400: Lower version of
aws_s3_v2_directory (1.2) of Atlas is is compatible with new Hooks
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73544/#review223398
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 1056 (patched)
<https://reviews.apache.org/r/73544/#comment312465>
replace line 1056-1058:
entities.addAll(context.getEntities())
also replace line 1062-1064:
entities.addAll(context.getReferredEntities().values())
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 1067 (patched)
<https://reviews.apache.org/r/73544/#comment312466>
'entities' can never be null, we assign it to empty List in line 1053. Consider checking if List is not empty.
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 1068 (patched)
<https://reviews.apache.org/r/73544/#comment312467>
for (AtlasEntity entity : entities) {
...
}
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
Lines 69 (patched)
<https://reviews.apache.org/r/73544/#comment312463>
S3V2_PREPROCESSOR_MAP => AWS_S3_V2_PREPROCESSOR_MAP
webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java
Lines 27 (patched)
<https://reviews.apache.org/r/73544/#comment312464>
S3V2Preprocessor => AWSS3V2DirectoryPreprocessor
webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java
Lines 56 (patched)
<https://reviews.apache.org/r/73544/#comment312469>
consider changing this to INFO log for debugging purpose.
webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java
Lines 66 (patched)
<https://reviews.apache.org/r/73544/#comment312468>
consider also including the 'name' attribute from aws_s3_v2_directory type in this check:
&& objectPrefix.contains(name)
qualifiedName : s3a://my_bucket_1/datalake/warehouse/tablespace/external/hive/employee.csv/@cm
objectPrefix : /datalake/warehouse/tablespace/external/hive/employee.csv/
name : employee.csv
- Sarath Subramanian
On Aug. 24, 2021, 11:21 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73544/
> -----------------------------------------------------------
>
> (Updated Aug. 24, 2021, 11:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4400
> https://issues.apache.org/jira/browse/ATLAS-4400
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4400: Fixed Hook and Atlas Preprocessor to handle S3 V2 directory objectPrefix Issue with Atlas Server and Hook versions mismatch
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java a9f2e5092
> common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 6bf5d5779
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 5643af95c
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 7f0cafe86
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 59f6440d5
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/S3V2Preprocessor.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/73544/diff/1/
>
>
> Testing
> -------
>
> Manually tested the Newer Atlas with old Hook and vice versa. Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/817/
>
>
> Thanks,
>
> Sidharth Mishra
>
>