You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Hemanth Yamijala <yh...@gmail.com> on 2016/06/06 14:27:47 UTC
Review Request 48280: ATLAS-515: Ability to initialize Kafka topics
with more than 1 replica
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/
-----------------------------------------------------------
Review request for atlas.
Bugs: ATLAS-515
https://issues.apache.org/jira/browse/ATLAS-515
Repository: atlas
Description
-------
Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
Diffs
-----
addons/falcon-bridge/pom.xml 14c6090
addons/hive-bridge/pom.xml 47e72e8
addons/storm-bridge/pom.xml b1a7a9b
distro/src/conf/atlas-application.properties 1cdd424
notification/pom.xml b3738db
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 71029b0
notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 1ee62d2
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc6
notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION
pom.xml eac95c8
Diff: https://reviews.apache.org/r/48280/diff/
Testing
-------
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly.
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation.
* Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued.
* Started 2 instances of Atlas server concurrently and verified topic creation is working fine.
* Existing UTs/ITs/new UTs pass.
Thanks,
Hemanth Yamijala
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka
topics with more than 1 replica
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/#review136932
-----------------------------------------------------------
Ship it!
Pls add documentation on how to use the scripts. The changes looks good to me.
- Suma Shivaprasad
On June 9, 2016, 10:30 a.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48280/
> -----------------------------------------------------------
>
> (Updated June 9, 2016, 10:30 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-515
> https://issues.apache.org/jira/browse/ATLAS-515
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_client_cmdline.py f109ad3
> distro/src/bin/atlas_config.py fab4046
> distro/src/bin/atlas_kafka_setup.py PRE-CREATION
> distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION
> distro/src/conf/atlas-application.properties 873c1e0
> distro/src/main/assemblies/standalone-package.xml 1c7b2c5
> notification/pom.xml b3738db
> notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION
> notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION
> pom.xml eac95c8
>
> Diff: https://reviews.apache.org/r/48280/diff/
>
>
> Testing
> -------
>
> * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly.
> * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation.
> * Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued.
> * Started 2 instances of Atlas server concurrently and verified topic creation is working fine.
> * Existing UTs/ITs/new UTs pass.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka
topics with more than 1 replica
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/#review137355
-----------------------------------------------------------
Ship it!
Ship It!
- Suma Shivaprasad
On June 13, 2016, 4:35 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48280/
> -----------------------------------------------------------
>
> (Updated June 13, 2016, 4:35 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-515
> https://issues.apache.org/jira/browse/ATLAS-515
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java 3dbab17
> distro/src/bin/atlas_client_cmdline.py f109ad3
> distro/src/bin/atlas_config.py fab4046
> distro/src/bin/atlas_kafka_setup.py PRE-CREATION
> distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION
> distro/src/conf/atlas-application.properties 2b90086
> distro/src/main/assemblies/standalone-package.xml 1c7b2c5
> docs/src/site/twiki/Configuration.twiki 7150483
> docs/src/site/twiki/InstallationSteps.twiki 518c380
> notification/pom.xml b3738db
> notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION
> notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION
> pom.xml e13345e
>
> Diff: https://reviews.apache.org/r/48280/diff/
>
>
> Testing
> -------
>
> * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly.
> * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation.
> * Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued.
> * Started 2 instances of Atlas server concurrently and verified topic creation is working fine.
> * Existing UTs/ITs/new UTs pass.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka
topics with more than 1 replica
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/
-----------------------------------------------------------
(Updated June 13, 2016, 4:35 p.m.)
Review request for atlas.
Changes
-------
Updated the patch to handle Kerberized Kafka clusters.
Bugs: ATLAS-515
https://issues.apache.org/jira/browse/ATLAS-515
Repository: atlas
Description
-------
Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java 3dbab17
distro/src/bin/atlas_client_cmdline.py f109ad3
distro/src/bin/atlas_config.py fab4046
distro/src/bin/atlas_kafka_setup.py PRE-CREATION
distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION
distro/src/conf/atlas-application.properties 2b90086
distro/src/main/assemblies/standalone-package.xml 1c7b2c5
docs/src/site/twiki/Configuration.twiki 7150483
docs/src/site/twiki/InstallationSteps.twiki 518c380
notification/pom.xml b3738db
notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION
notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION
pom.xml e13345e
Diff: https://reviews.apache.org/r/48280/diff/
Testing
-------
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly.
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation.
* Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued.
* Started 2 instances of Atlas server concurrently and verified topic creation is working fine.
* Existing UTs/ITs/new UTs pass.
Thanks,
Hemanth Yamijala
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka
topics with more than 1 replica
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/
-----------------------------------------------------------
(Updated June 10, 2016, 8:44 a.m.)
Review request for atlas.
Changes
-------
The new patch adds documentation to Installation and Configuration pages for using the added scripts.
Bugs: ATLAS-515
https://issues.apache.org/jira/browse/ATLAS-515
Repository: atlas
Description
-------
Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
Diffs (updated)
-----
distro/src/bin/atlas_client_cmdline.py f109ad3
distro/src/bin/atlas_config.py fab4046
distro/src/bin/atlas_kafka_setup.py PRE-CREATION
distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION
distro/src/conf/atlas-application.properties 873c1e0
distro/src/main/assemblies/standalone-package.xml 1c7b2c5
docs/src/site/twiki/Configuration.twiki 7150483
docs/src/site/twiki/InstallationSteps.twiki 9699a94
notification/pom.xml b3738db
notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION
notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION
pom.xml eac95c8
Diff: https://reviews.apache.org/r/48280/diff/
Testing
-------
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly.
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation.
* Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued.
* Started 2 instances of Atlas server concurrently and verified topic creation is working fine.
* Existing UTs/ITs/new UTs pass.
Thanks,
Hemanth Yamijala
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka
topics with more than 1 replica
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/
-----------------------------------------------------------
(Updated June 9, 2016, 10:30 a.m.)
Review request for atlas.
Changes
-------
As described in this [comment](https://issues.apache.org/jira/browse/ATLAS-515?focusedCommentId=15320301&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15320301) on ATLAS-515, I am changing the approach to initializing Kafka topics from being automated to being script driven, where the script can be automated by installation tools like Ambari.
Changes in this patch:
* Retains KafkaTopicCreator class with the logic for setting up topics
* Two python scripts - atlas_kafka_setup.py and atlas_kafka_setup_hook.py.
* Changes to pom.xml to include the required dependencies
Bugs: ATLAS-515
https://issues.apache.org/jira/browse/ATLAS-515
Repository: atlas
Description
-------
Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
Diffs (updated)
-----
distro/src/bin/atlas_client_cmdline.py f109ad3
distro/src/bin/atlas_config.py fab4046
distro/src/bin/atlas_kafka_setup.py PRE-CREATION
distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION
distro/src/conf/atlas-application.properties 873c1e0
distro/src/main/assemblies/standalone-package.xml 1c7b2c5
notification/pom.xml b3738db
notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION
notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION
pom.xml eac95c8
Diff: https://reviews.apache.org/r/48280/diff/
Testing
-------
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly.
* Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation.
* Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued.
* Started 2 instances of Atlas server concurrently and verified topic creation is working fine.
* Existing UTs/ITs/new UTs pass.
Thanks,
Hemanth Yamijala