You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2023/03/14 15:35:03 UTC
Review Request 74348: RANGER-4130: Improve performance of event processing in agsync by optimizing number of commits to Kafka broker
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74348/
-----------------------------------------------------------
Review request for ranger, madhan, Madhan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-4130
https://issues.apache.org/jira/browse/RANGER-4130
Repository: ranger
Description
-------
tagsync, currently, commits to Kafka every handled event after successfully uploading to Ranger admin. The commits are synchronous and may impose performance penalty.
This Jira is to update event handling in tagsync by optmizing on the commits to Kafka while reducing the chances of Kafka delivering same message multiple times to tagsync.
This Jira also fixes a potential issue if the Kafka queue contains only a large number of notifications not handled by tagsync. In such cases, tagsync does not do any commit to kafka, and that may cause back-pressure on Kafka queue.
Diffs
-----
tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java a618cc986
Diff: https://reviews.apache.org/r/74348/diff/1/
Testing
-------
Compiled and ran unit tests cleanly
Thanks,
Abhay Kulkarni
Re: Review Request 74348: RANGER-4130: Improve performance of event processing in agsync by optimizing number of commits to Kafka broker
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74348/#review225266
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On March 14, 2023, 3:35 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74348/
> -----------------------------------------------------------
>
> (Updated March 14, 2023, 3:35 p.m.)
>
>
> Review request for ranger, madhan, Madhan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-4130
> https://issues.apache.org/jira/browse/RANGER-4130
>
>
> Repository: ranger
>
>
> Description
> -------
>
> tagsync, currently, commits to Kafka every handled event after successfully uploading to Ranger admin. The commits are synchronous and may impose performance penalty.
>
> This Jira is to update event handling in tagsync by optmizing on the commits to Kafka while reducing the chances of Kafka delivering same message multiple times to tagsync.
>
> This Jira also fixes a potential issue if the Kafka queue contains only a large number of notifications not handled by tagsync. In such cases, tagsync does not do any commit to kafka, and that may cause back-pressure on Kafka queue.
>
>
> Diffs
> -----
>
> tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java a618cc986
>
>
> Diff: https://reviews.apache.org/r/74348/diff/1/
>
>
> Testing
> -------
>
> Compiled and ran unit tests cleanly
>
>
> Thanks,
>
> Abhay Kulkarni
>
>