You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Deep Singh <de...@gmail.com> on 2020/11/30 19:06:48 UTC

Review Request 73049: ATLAS-4044: Atlas import-kafka.sh does not import "__consumer_offsets" topic

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.


Bugs: ATLAS-4044
    https://issues.apache.org/jira/browse/ATLAS-4044


Repository: atlas


Description
-------

Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
With this fix I have introduced an API option to include the internal topic as well.


Diffs
-----

  common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 


Diff: https://reviews.apache.org/r/73049/diff/1/


Testing
-------

Manual testing done.


Thanks,

Deep Singh


Re: Review Request 73049: ATLAS-4044: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222256
-----------------------------------------------------------



Please update the JIRA referenced in this review; it should be ATLAS-4043.


common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
Line 109 (original), 104 (patched)
<https://reviews.apache.org/r/73049/#comment311324>

    Tracking Kafka's internal topics in Atlas may not be useful. I suggest to introduce a config to control whether to import internal topics or not.
     atlas.hook.kafka.import.internal.topics
    
    The default should be false; deployments that need to track internal topics can set this configuration to true in atlas-application.properties.


- Madhan Neethiraj


On Nov. 30, 2020, 7:06 p.m., Deep Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
> 
> (Updated Nov. 30, 2020, 7:06 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4044
>     https://issues.apache.org/jira/browse/ATLAS-4044
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
> With this fix I have introduced an API option to include the internal topic as well.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 
> 
> 
> Diff: https://reviews.apache.org/r/73049/diff/1/
> 
> 
> Testing
> -------
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>


Re: Review Request 73049: ATLAS-4044: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222252
-----------------------------------------------------------


Fix it, then Ship it!





common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
Line 25 (original), 25 (patched)
<https://reviews.apache.org/r/73049/#comment311319>

    consider avoiding wildcard imports and explicitly listi imports to avoid cluttering namespace.


- Sarath Subramanian


On Nov. 30, 2020, 11:06 a.m., Deep Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
> 
> (Updated Nov. 30, 2020, 11:06 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4044
>     https://issues.apache.org/jira/browse/ATLAS-4044
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
> With this fix I have introduced an API option to include the internal topic as well.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 
> 
> 
> Diff: https://reviews.apache.org/r/73049/diff/1/
> 
> 
> Testing
> -------
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>


Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Deep Singh <de...@gmail.com>.

> On Dec. 1, 2020, 7:20 p.m., Sarath Subramanian wrote:
> > common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
> > Lines 61 (patched)
> > <https://reviews.apache.org/r/73049/diff/2/?file=2242526#file2242526line61>
> >
> >     consider renaming to:
> >     
> >     "atlas.hook.kafka.import.internal.topics" => "atlas.kafka.bridge.enable.internal.topics.import"

I some how missed this comment. This patch is already submitted. I will open another ticket to fix this.


- Deep


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222259
-----------------------------------------------------------


On Dec. 1, 2020, 6:02 p.m., Deep Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
> 
> (Updated Dec. 1, 2020, 6:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4043
>     https://issues.apache.org/jira/browse/ATLAS-4043
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
> With this fix I have introduced an API option to include the internal topic as well.
> The option is configuration driven. 
> One need to set atlas.hook.kafka.import.internal.topics=true in the atlas application properties(/etc/atlas/conf/atlas-application.properties)
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 
> 
> 
> Diff: https://reviews.apache.org/r/73049/diff/2/
> 
> 
> Testing
> -------
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>


Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222259
-----------------------------------------------------------


Fix it, then Ship it!





common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
Lines 61 (patched)
<https://reviews.apache.org/r/73049/#comment311326>

    consider renaming to:
    
    "atlas.hook.kafka.import.internal.topics" => "atlas.kafka.bridge.enable.internal.topics.import"


- Sarath Subramanian


On Dec. 1, 2020, 10:02 a.m., Deep Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
> 
> (Updated Dec. 1, 2020, 10:02 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4043
>     https://issues.apache.org/jira/browse/ATLAS-4043
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
> With this fix I have introduced an API option to include the internal topic as well.
> The option is configuration driven. 
> One need to set atlas.hook.kafka.import.internal.topics=true in the atlas application properties(/etc/atlas/conf/atlas-application.properties)
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 
> 
> 
> Diff: https://reviews.apache.org/r/73049/diff/2/
> 
> 
> Testing
> -------
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>


Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Deep Singh <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------

(Updated Dec. 1, 2020, 6:02 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.


Bugs: ATLAS-4043
    https://issues.apache.org/jira/browse/ATLAS-4043


Repository: atlas


Description (updated)
-------

Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
With this fix I have introduced an API option to include the internal topic as well.
The option is configuration driven. 
One need to set atlas.hook.kafka.import.internal.topics=true in the atlas application properties(/etc/atlas/conf/atlas-application.properties)


Diffs
-----

  common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 


Diff: https://reviews.apache.org/r/73049/diff/2/


Testing
-------

Manual testing done.


Thanks,

Deep Singh


Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222258
-----------------------------------------------------------


Ship it!




Ship It!

- Madhan Neethiraj


On Dec. 1, 2020, 5:51 p.m., Deep Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
> 
> (Updated Dec. 1, 2020, 5:51 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4043
>     https://issues.apache.org/jira/browse/ATLAS-4043
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
> With this fix I have introduced an API option to include the internal topic as well.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 
> 
> 
> Diff: https://reviews.apache.org/r/73049/diff/2/
> 
> 
> Testing
> -------
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>


Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Deep Singh <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------

(Updated Dec. 1, 2020, 5:51 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.


Bugs: ATLAS-4043
    https://issues.apache.org/jira/browse/ATLAS-4043


Repository: atlas


Description
-------

Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
With this fix I have introduced an API option to include the internal topic as well.


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 


Diff: https://reviews.apache.org/r/73049/diff/2/

Changes: https://reviews.apache.org/r/73049/diff/1-2/


Testing
-------

Manual testing done.


Thanks,

Deep Singh


Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic

Posted by Deep Singh <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------

(Updated Dec. 1, 2020, 4:16 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.


Summary (updated)
-----------------

ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic


Bugs: ATLAS-4043
    https://issues.apache.org/jira/browse/ATLAS-4043


Repository: atlas


Description
-------

Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka. 
With this fix I have introduced an API option to include the internal topic as well.


Diffs
-----

  common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b 


Diff: https://reviews.apache.org/r/73049/diff/1/


Testing
-------

Manual testing done.


Thanks,

Deep Singh