You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ramesh Mani <rm...@hortonworks.com> on 2018/04/19 19:12:19 UTC
Review Request 66721: ATLAS-2586:Provide a utility to Import Kafka
Entities into Atlas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66721/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sharmadha Sainath.
Repository: atlas
Description
-------
ATLAS-2586:Provide a utility to Import Kafka Entities into Atlas
Diffs
-----
addons/kafka-bridge/pom.xml PRE-CREATION
addons/kafka-bridge/src/bin/import-kafka.sh PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java PRE-CREATION
addons/kafka-bridge/src/main/resources/atlas-kafka-import-log4j.xml PRE-CREATION
addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json PRE-CREATION
distro/pom.xml 1f4c6d5
distro/src/main/assemblies/atlas-kafka-hook-package.xml PRE-CREATION
distro/src/main/assemblies/standalone-package.xml dc2a66b
pom.xml a6d1268
Diff: https://reviews.apache.org/r/66721/diff/1/
Testing
-------
Verified in local secure cluster.
Usage 1: import-kafka.sh [-n <topic regex> OR --topic <topic regex >]
Usage 2: import-kafka.sh [-f <filename>]
Format:
topic1
topic2
topic3
Thanks,
Ramesh Mani
Re: Review Request 66721: ATLAS-2586:Provide a utility to Import Kafka
Entities into Atlas
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66721/#review201550
-----------------------------------------------------------
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 73 (patched)
<https://reviews.apache.org/r/66721/#comment282846>
KAKFA_TOPIC_QUALIFIED_NAME ==> FORMAT_KAKFA_TOPIC_QUALIFIED_NAME
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 159 (patched)
<https://reviews.apache.org/r/66721/#comment282849>
ImportKafkaEntities() ==> importTopics()
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java
Lines 27 (patched)
<https://reviews.apache.org/r/66721/#comment282855>
If KAFKA_PARTITION is not being handled currently, I would suggest removing this.
addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json
Lines 14 (patched)
<https://reviews.apache.org/r/66721/#comment282854>
Please mark 'isIndexable' as true for: partitionCount, createTime, modifiedTime
pom.xml
Lines 1499 (patched)
<https://reviews.apache.org/r/66721/#comment282845>
Artifact kafka-bridge-shim is not built; please review and remove this dependency.
- Madhan Neethiraj
On April 19, 2018, 7:12 p.m., Ramesh Mani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66721/
> -----------------------------------------------------------
>
> (Updated April 19, 2018, 7:12 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sharmadha Sainath.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-2586:Provide a utility to Import Kafka Entities into Atlas
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml PRE-CREATION
> addons/kafka-bridge/src/bin/import-kafka.sh PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java PRE-CREATION
> addons/kafka-bridge/src/main/resources/atlas-kafka-import-log4j.xml PRE-CREATION
> addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json PRE-CREATION
> distro/pom.xml 1f4c6d5
> distro/src/main/assemblies/atlas-kafka-hook-package.xml PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml dc2a66b
> pom.xml a6d1268
>
>
> Diff: https://reviews.apache.org/r/66721/diff/1/
>
>
> Testing
> -------
>
> Verified in local secure cluster.
>
> Usage 1: import-kafka.sh [-n <topic regex> OR --topic <topic regex >]
> Usage 2: import-kafka.sh [-f <filename>]
> Format:
> topic1
> topic2
> topic3
>
>
> Thanks,
>
> Ramesh Mani
>
>
Re: Review Request 66721: ATLAS-2586:Provide a utility to Import Kafka
Entities into Atlas
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66721/#review201585
-----------------------------------------------------------
Fix it, then Ship it!
addons/kafka-bridge/pom.xml
Lines 45 (patched)
<https://reviews.apache.org/r/66721/#comment282906>
atlas-notification library may not be necessary for a bridge. Please review.
Also, review other dependencies listed here; few of them might be required only in test - like hadoop-minicluster; and some of them may not be necessary - like hdfs-model, atlas-notification.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 160 (patched)
<https://reviews.apache.org/r/66721/#comment282905>
Consider renaming this as 'availableTopics' and initialize in the constructor - to avoid calling zkUtils.getAllTopics() multiple times in case of processing topics list from a file.
- Madhan Neethiraj
On April 19, 2018, 9:56 p.m., Ramesh Mani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66721/
> -----------------------------------------------------------
>
> (Updated April 19, 2018, 9:56 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sharmadha Sainath.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-2586:Provide a utility to Import Kafka Entities into Atlas
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml PRE-CREATION
> addons/kafka-bridge/src/bin/import-kafka.sh PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java PRE-CREATION
> addons/kafka-bridge/src/main/resources/atlas-kafka-import-log4j.xml PRE-CREATION
> addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json PRE-CREATION
> distro/pom.xml 1f4c6d5
> distro/src/main/assemblies/atlas-kafka-hook-package.xml PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml dc2a66b
> pom.xml 9c9d746
>
>
> Diff: https://reviews.apache.org/r/66721/diff/2/
>
>
> Testing
> -------
>
> Verified in local secure cluster.
>
> Usage 1: import-kafka.sh [-n <topic regex> OR --topic <topic regex >]
> Usage 2: import-kafka.sh [-f <filename>]
> Format:
> topic1
> topic2
> topic3
>
>
> Thanks,
>
> Ramesh Mani
>
>
Re: Review Request 66721: ATLAS-2586:Provide a utility to Import Kafka
Entities into Atlas
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66721/#review201648
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On April 20, 2018, 5:59 a.m., Ramesh Mani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66721/
> -----------------------------------------------------------
>
> (Updated April 20, 2018, 5:59 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sharmadha Sainath.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-2586:Provide a utility to Import Kafka Entities into Atlas
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml PRE-CREATION
> addons/kafka-bridge/src/bin/import-kafka.sh PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java PRE-CREATION
> addons/kafka-bridge/src/main/resources/atlas-kafka-import-log4j.xml PRE-CREATION
> addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json PRE-CREATION
> distro/pom.xml 1f4c6d5
> distro/src/main/assemblies/atlas-kafka-hook-package.xml PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml dc2a66b
> pom.xml 9c9d746
>
>
> Diff: https://reviews.apache.org/r/66721/diff/3/
>
>
> Testing
> -------
>
> Verified in local secure cluster.
>
> Usage 1: import-kafka.sh [-n <topic regex> OR --topic <topic regex >]
> Usage 2: import-kafka.sh [-f <filename>]
> Format:
> topic1
> topic2
> topic3
>
>
> Thanks,
>
> Ramesh Mani
>
>
Re: Review Request 66721: ATLAS-2586:Provide a utility to Import Kafka
Entities into Atlas
Posted by Ramesh Mani <rm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66721/
-----------------------------------------------------------
(Updated April 20, 2018, 5:59 a.m.)
Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sharmadha Sainath.
Changes
-------
Fixed review comments
Repository: atlas
Description
-------
ATLAS-2586:Provide a utility to Import Kafka Entities into Atlas
Diffs (updated)
-----
addons/kafka-bridge/pom.xml PRE-CREATION
addons/kafka-bridge/src/bin/import-kafka.sh PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java PRE-CREATION
addons/kafka-bridge/src/main/resources/atlas-kafka-import-log4j.xml PRE-CREATION
addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json PRE-CREATION
distro/pom.xml 1f4c6d5
distro/src/main/assemblies/atlas-kafka-hook-package.xml PRE-CREATION
distro/src/main/assemblies/standalone-package.xml dc2a66b
pom.xml 9c9d746
Diff: https://reviews.apache.org/r/66721/diff/3/
Changes: https://reviews.apache.org/r/66721/diff/2-3/
Testing
-------
Verified in local secure cluster.
Usage 1: import-kafka.sh [-n <topic regex> OR --topic <topic regex >]
Usage 2: import-kafka.sh [-f <filename>]
Format:
topic1
topic2
topic3
Thanks,
Ramesh Mani
Re: Review Request 66721: ATLAS-2586:Provide a utility to Import Kafka
Entities into Atlas
Posted by Ramesh Mani <rm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66721/
-----------------------------------------------------------
(Updated April 19, 2018, 9:56 p.m.)
Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sharmadha Sainath.
Changes
-------
Fixed review comments
Repository: atlas
Description
-------
ATLAS-2586:Provide a utility to Import Kafka Entities into Atlas
Diffs (updated)
-----
addons/kafka-bridge/pom.xml PRE-CREATION
addons/kafka-bridge/src/bin/import-kafka.sh PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java PRE-CREATION
addons/kafka-bridge/src/main/resources/atlas-kafka-import-log4j.xml PRE-CREATION
addons/models/1000-Hadoop/patches/006-kafka_topic_add_attribute.json PRE-CREATION
distro/pom.xml 1f4c6d5
distro/src/main/assemblies/atlas-kafka-hook-package.xml PRE-CREATION
distro/src/main/assemblies/standalone-package.xml dc2a66b
pom.xml 9c9d746
Diff: https://reviews.apache.org/r/66721/diff/2/
Changes: https://reviews.apache.org/r/66721/diff/1-2/
Testing
-------
Verified in local secure cluster.
Usage 1: import-kafka.sh [-n <topic regex> OR --topic <topic regex >]
Usage 2: import-kafka.sh [-f <filename>]
Format:
topic1
topic2
topic3
Thanks,
Ramesh Mani