You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Deep Singh <de...@gmail.com> on 2020/11/30 19:06:48 UTC
Review Request 73049: ATLAS-4044: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-4044
https://issues.apache.org/jira/browse/ATLAS-4044
Repository: atlas
Description
-------
Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
With this fix I have introduced an API option to include the internal topic as well.
Diffs
-----
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
Diff: https://reviews.apache.org/r/73049/diff/1/
Testing
-------
Manual testing done.
Thanks,
Deep Singh
Re: Review Request 73049: ATLAS-4044: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222256
-----------------------------------------------------------
Please update the JIRA referenced in this review; it should be ATLAS-4043.
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
Line 109 (original), 104 (patched)
<https://reviews.apache.org/r/73049/#comment311324>
Tracking Kafka's internal topics in Atlas may not be useful. I suggest to introduce a config to control whether to import internal topics or not.
atlas.hook.kafka.import.internal.topics
The default should be false; deployments that need to track internal topics can set this configuration to true in atlas-application.properties.
- Madhan Neethiraj
On Nov. 30, 2020, 7:06 p.m., Deep Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2020, 7:06 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4044
> https://issues.apache.org/jira/browse/ATLAS-4044
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
> With this fix I have introduced an API option to include the internal topic as well.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
>
>
> Diff: https://reviews.apache.org/r/73049/diff/1/
>
>
> Testing
> -------
>
> Manual testing done.
>
>
> Thanks,
>
> Deep Singh
>
>
Re: Review Request 73049: ATLAS-4044: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222252
-----------------------------------------------------------
Fix it, then Ship it!
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
Line 25 (original), 25 (patched)
<https://reviews.apache.org/r/73049/#comment311319>
consider avoiding wildcard imports and explicitly listi imports to avoid cluttering namespace.
- Sarath Subramanian
On Nov. 30, 2020, 11:06 a.m., Deep Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2020, 11:06 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4044
> https://issues.apache.org/jira/browse/ATLAS-4044
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
> With this fix I have introduced an API option to include the internal topic as well.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
>
>
> Diff: https://reviews.apache.org/r/73049/diff/1/
>
>
> Testing
> -------
>
> Manual testing done.
>
>
> Thanks,
>
> Deep Singh
>
>
Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Deep Singh <de...@gmail.com>.
> On Dec. 1, 2020, 7:20 p.m., Sarath Subramanian wrote:
> > common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
> > Lines 61 (patched)
> > <https://reviews.apache.org/r/73049/diff/2/?file=2242526#file2242526line61>
> >
> > consider renaming to:
> >
> > "atlas.hook.kafka.import.internal.topics" => "atlas.kafka.bridge.enable.internal.topics.import"
I some how missed this comment. This patch is already submitted. I will open another ticket to fix this.
- Deep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222259
-----------------------------------------------------------
On Dec. 1, 2020, 6:02 p.m., Deep Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2020, 6:02 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4043
> https://issues.apache.org/jira/browse/ATLAS-4043
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
> With this fix I have introduced an API option to include the internal topic as well.
> The option is configuration driven.
> One need to set atlas.hook.kafka.import.internal.topics=true in the atlas application properties(/etc/atlas/conf/atlas-application.properties)
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
>
>
> Diff: https://reviews.apache.org/r/73049/diff/2/
>
>
> Testing
> -------
>
> Manual testing done.
>
>
> Thanks,
>
> Deep Singh
>
>
Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222259
-----------------------------------------------------------
Fix it, then Ship it!
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
Lines 61 (patched)
<https://reviews.apache.org/r/73049/#comment311326>
consider renaming to:
"atlas.hook.kafka.import.internal.topics" => "atlas.kafka.bridge.enable.internal.topics.import"
- Sarath Subramanian
On Dec. 1, 2020, 10:02 a.m., Deep Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2020, 10:02 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4043
> https://issues.apache.org/jira/browse/ATLAS-4043
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
> With this fix I have introduced an API option to include the internal topic as well.
> The option is configuration driven.
> One need to set atlas.hook.kafka.import.internal.topics=true in the atlas application properties(/etc/atlas/conf/atlas-application.properties)
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
>
>
> Diff: https://reviews.apache.org/r/73049/diff/2/
>
>
> Testing
> -------
>
> Manual testing done.
>
>
> Thanks,
>
> Deep Singh
>
>
Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Deep Singh <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------
(Updated Dec. 1, 2020, 6:02 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-4043
https://issues.apache.org/jira/browse/ATLAS-4043
Repository: atlas
Description (updated)
-------
Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
With this fix I have introduced an API option to include the internal topic as well.
The option is configuration driven.
One need to set atlas.hook.kafka.import.internal.topics=true in the atlas application properties(/etc/atlas/conf/atlas-application.properties)
Diffs
-----
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
Diff: https://reviews.apache.org/r/73049/diff/2/
Testing
-------
Manual testing done.
Thanks,
Deep Singh
Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/#review222258
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Dec. 1, 2020, 5:51 p.m., Deep Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73049/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2020, 5:51 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4043
> https://issues.apache.org/jira/browse/ATLAS-4043
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
> With this fix I have introduced an API option to include the internal topic as well.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
>
>
> Diff: https://reviews.apache.org/r/73049/diff/2/
>
>
> Testing
> -------
>
> Manual testing done.
>
>
> Thanks,
>
> Deep Singh
>
>
Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Deep Singh <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------
(Updated Dec. 1, 2020, 5:51 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-4043
https://issues.apache.org/jira/browse/ATLAS-4043
Repository: atlas
Description
-------
Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
With this fix I have introduced an API option to include the internal topic as well.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
Diff: https://reviews.apache.org/r/73049/diff/2/
Changes: https://reviews.apache.org/r/73049/diff/1-2/
Testing
-------
Manual testing done.
Thanks,
Deep Singh
Re: Review Request 73049: ATLAS-4043: Atlas import-kafka.sh does not
import "__consumer_offsets" topic
Posted by Deep Singh <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73049/
-----------------------------------------------------------
(Updated Dec. 1, 2020, 4:16 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian.
Summary (updated)
-----------------
ATLAS-4043: Atlas import-kafka.sh does not import "__consumer_offsets" topic
Bugs: ATLAS-4043
https://issues.apache.org/jira/browse/ATLAS-4043
Repository: atlas
Description
-------
Recently changes were made to remove dependency on zookeeper. This bug got introduced as regression. Kafka AdminClient is now used to get the list of existing topics. The API removes the topic which are internal to Kafka.
With this fix I have introduced an API option to include the internal topic as well.
Diffs
-----
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 7a397b19b
Diff: https://reviews.apache.org/r/73049/diff/1/
Testing
-------
Manual testing done.
Thanks,
Deep Singh