You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2023/03/10 00:17:55 UTC
Review Request 74344: RANGER-4129: ArrayIndexOutOfBounds exception may be thrown while processing events
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74344/
-----------------------------------------------------------
Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.
Bugs: RANGER-4129
https://issues.apache.org/jira/browse/RANGER-4129
Repository: ranger
Description
-------
If AtlasTagSource.buildAndUploadServiceTags() is called with empty AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds exception is thrown when AtlasTagSource.messages list is read. This may happen when the first notification processed by tagsync process is of type ENTITY_DELETE.
This Jira also fixes a potential issue if the Kafka queue contains only a large number of notifications not expected to be handled by tagsync. In such cases, tagsync will not acknowledge receipt and it may cause back-pressure on Kafka queue.
Diffs
-----
tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java a7c456b3d
tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java 1a3ddecb5
Diff: https://reviews.apache.org/r/74344/diff/1/
Testing
-------
Compiled clean and passed all unit tests
Thanks,
Abhay Kulkarni
Re: Review Request 74344: RANGER-4129: ArrayIndexOutOfBounds exception may be thrown while processing events
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74344/#review225261
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On March 11, 2023, 2:32 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74344/
> -----------------------------------------------------------
>
> (Updated March 11, 2023, 2:32 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-4129
> https://issues.apache.org/jira/browse/RANGER-4129
>
>
> Repository: ranger
>
>
> Description
> -------
>
> If AtlasTagSource.buildAndUploadServiceTags() is called with empty AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds exception is thrown when AtlasTagSource.messages list is read. This may happen when the first notification processed by tagsync process is of type ENTITY_DELETE.
>
>
> Diffs
> -----
>
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java a7c456b3d
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java 1a3ddecb5
>
>
> Diff: https://reviews.apache.org/r/74344/diff/2/
>
>
> Testing
> -------
>
> Compiled clean and passed all unit tests
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 74344: RANGER-4129: ArrayIndexOutOfBounds exception may be thrown while processing events
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74344/
-----------------------------------------------------------
(Updated March 11, 2023, 2:32 p.m.)
Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.
Changes
-------
Patch and JIRA description updated to address only the ArrayIndexOutOfBounds condition.
Bugs: RANGER-4129
https://issues.apache.org/jira/browse/RANGER-4129
Repository: ranger
Description (updated)
-------
If AtlasTagSource.buildAndUploadServiceTags() is called with empty AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds exception is thrown when AtlasTagSource.messages list is read. This may happen when the first notification processed by tagsync process is of type ENTITY_DELETE.
Diffs (updated)
-----
tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java a7c456b3d
tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java 1a3ddecb5
Diff: https://reviews.apache.org/r/74344/diff/2/
Changes: https://reviews.apache.org/r/74344/diff/1-2/
Testing
-------
Compiled clean and passed all unit tests
Thanks,
Abhay Kulkarni
Re: Review Request 74344: RANGER-4129: ArrayIndexOutOfBounds exception may be thrown while processing events
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
> On March 10, 2023, 2:24 a.m., Madhan Neethiraj wrote:
> > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java
> > Lines 353 (patched)
> > <https://reviews.apache.org/r/74344/diff/1/?file=2274915#file2274915line357>
> >
> > Is it necessary to commit each unhandled message? Wouldn't committing the last message be enough?
> >
> > If that works, it would be necessary to only hold the last message in memory i.e. eliminate unhandledMessages. Also, commitToKafka() should clear lastUnhandledMessage - as it shouldn't be call commit on earlier message offset.
tagsync supports a configuration parameter for specifying the size of the batch of messages read from the Kafka queue (which may be greater than 1) and then processed. In that case, the batch of messages being may contain a mix of handled and unhandled messages and each message has to be committed individually.
- Abhay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74344/#review225258
-----------------------------------------------------------
On March 10, 2023, 12:17 a.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74344/
> -----------------------------------------------------------
>
> (Updated March 10, 2023, 12:17 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-4129
> https://issues.apache.org/jira/browse/RANGER-4129
>
>
> Repository: ranger
>
>
> Description
> -------
>
> If AtlasTagSource.buildAndUploadServiceTags() is called with empty AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds exception is thrown when AtlasTagSource.messages list is read. This may happen when the first notification processed by tagsync process is of type ENTITY_DELETE.
>
> This Jira also fixes a potential issue if the Kafka queue contains only a large number of notifications not expected to be handled by tagsync. In such cases, tagsync will not acknowledge receipt and it may cause back-pressure on Kafka queue.
>
>
> Diffs
> -----
>
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java a7c456b3d
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java 1a3ddecb5
>
>
> Diff: https://reviews.apache.org/r/74344/diff/1/
>
>
> Testing
> -------
>
> Compiled clean and passed all unit tests
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 74344: RANGER-4129: ArrayIndexOutOfBounds exception may be thrown while processing events
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
> On March 10, 2023, 2:24 a.m., Madhan Neethiraj wrote:
> > tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java
> > Lines 353 (patched)
> > <https://reviews.apache.org/r/74344/diff/1/?file=2274915#file2274915line357>
> >
> > Is it necessary to commit each unhandled message? Wouldn't committing the last message be enough?
> >
> > If that works, it would be necessary to only hold the last message in memory i.e. eliminate unhandledMessages. Also, commitToKafka() should clear lastUnhandledMessage - as it shouldn't be call commit on earlier message offset.
>
> Abhay Kulkarni wrote:
> tagsync supports a configuration parameter for specifying the size of the batch of messages read from the Kafka queue (which may be greater than 1) and then processed. In that case, the batch of messages being may contain a mix of handled and unhandled messages and each message has to be committed individually.
This fix now only addresses the Array out-of-bounds issue. For optimizing the commits to Kafka, another JIRA (RANGER-4130) is created.
- Abhay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74344/#review225258
-----------------------------------------------------------
On March 10, 2023, 12:17 a.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74344/
> -----------------------------------------------------------
>
> (Updated March 10, 2023, 12:17 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-4129
> https://issues.apache.org/jira/browse/RANGER-4129
>
>
> Repository: ranger
>
>
> Description
> -------
>
> If AtlasTagSource.buildAndUploadServiceTags() is called with empty AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds exception is thrown when AtlasTagSource.messages list is read. This may happen when the first notification processed by tagsync process is of type ENTITY_DELETE.
>
> This Jira also fixes a potential issue if the Kafka queue contains only a large number of notifications not expected to be handled by tagsync. In such cases, tagsync will not acknowledge receipt and it may cause back-pressure on Kafka queue.
>
>
> Diffs
> -----
>
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java a7c456b3d
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java 1a3ddecb5
>
>
> Diff: https://reviews.apache.org/r/74344/diff/1/
>
>
> Testing
> -------
>
> Compiled clean and passed all unit tests
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 74344: RANGER-4129: ArrayIndexOutOfBounds exception may be thrown while processing events
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74344/#review225258
-----------------------------------------------------------
tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java
Lines 353 (patched)
<https://reviews.apache.org/r/74344/#comment313978>
Is it necessary to commit each unhandled message? Wouldn't committing the last message be enough?
If that works, it would be necessary to only hold the last message in memory i.e. eliminate unhandledMessages. Also, commitToKafka() should clear lastUnhandledMessage - as it shouldn't be call commit on earlier message offset.
- Madhan Neethiraj
On March 10, 2023, 12:17 a.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74344/
> -----------------------------------------------------------
>
> (Updated March 10, 2023, 12:17 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, madhan, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-4129
> https://issues.apache.org/jira/browse/RANGER-4129
>
>
> Repository: ranger
>
>
> Description
> -------
>
> If AtlasTagSource.buildAndUploadServiceTags() is called with empty AtlasTagSource.atlasEntityWithTags list, then an ArrayIndexOutOfBounds exception is thrown when AtlasTagSource.messages list is read. This may happen when the first notification processed by tagsync process is of type ENTITY_DELETE.
>
> This Jira also fixes a potential issue if the Kafka queue contains only a large number of notifications not expected to be handled by tagsync. In such cases, tagsync will not acknowledge receipt and it may cause back-pressure on Kafka queue.
>
>
> Diffs
> -----
>
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasNotificationMapper.java a7c456b3d
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java 1a3ddecb5
>
>
> Diff: https://reviews.apache.org/r/74344/diff/1/
>
>
> Testing
> -------
>
> Compiled clean and passed all unit tests
>
>
> Thanks,
>
> Abhay Kulkarni
>
>