You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Ambud Sharma <as...@gmail.com> on 2017/06/20 21:09:41 UTC
Review Request 60249: Ambari rack awareness for Kafka
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/
-----------------------------------------------------------
Review request for Ambari.
Repository: ambari
Description
-------
Ambari rack awareness for Kafka
Diffs
-----
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py 680dd32802
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py 9acc1ef203
Diff: https://reviews.apache.org/r/60249/diff/1/
Testing
-------
Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
Thanks,
Ambud Sharma
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Ambud Sharma <as...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/
-----------------------------------------------------------
(Updated June 21, 2017, 8:33 p.m.)
Review request for Ambari and Alejandro Fernandez.
Changes
-------
Added description and reviewer
Repository: ambari
Description
-------
Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
Design:
This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/kafka.py 680dd32802
ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py 5b0be54982
Diff: https://reviews.apache.org/r/60249/diff/2/
Changes: https://reviews.apache.org/r/60249/diff/1-2/
Testing
-------
Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
Thanks,
Ambud Sharma
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Sebastian Toader <st...@hortonworks.com>.
> On June 21, 2017, 8:04 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
> > Lines 106-115 (patched)
> > <https://reviews.apache.org/r/60249/diff/1/?file=1754688#file1754688line106>
> >
> > The ```params.all_racks``` provides only access to rack info that was set up for hosts within ambari. However there rack info may come from a custom script in that case the information in ```params.all_racks``` won't have correct data.
> >
> > Use the topology script file to figure out the rack information for the current cluster. The path to the topology script is found in ```core-site/net.topology.script.file.name```
> >
> > The topology script file accepts one or more host fqdn as input parameter and will return to stdout the rack info for the given hosts.
>
> Ambud Sharma wrote:
> The use case here is to populate information of Node rack in Kafka configuration. Unlike Hadoop (HDFS/YARN) where the topology script is applicable and supported natively, Kafka takes rack information differently i.e. via a property file. While this could be an enhancement later, most users simply want to propagate rack information for a given Kafka broker in Ambari and publish that to Kafka.
>
> core-site does not get installed (there is no /etc/hadoop folder) when you install Kafka, the topology mapping file does not get published; we shouldn't make this a dependency or publish a file that creates confusion for the standard Kafka configuration file set.
The ```topology_script.py``` is generated by BEFORE-START hook (see ```/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py```), this ensures that the file is present on all nodes.
Note that core-site.xml that has the path to the ```topology_script.py``` is created only of at least one HDP component that requires it is installed to the host. To ensure that core-site.xml is present for Kafka as well you'll need to add core-site to metainfo.xml/<configuration-dependencies>
- Sebastian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/#review178450
-----------------------------------------------------------
On June 21, 2017, 10:33 p.m., Ambud Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60249/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 10:33 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
>
> As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
> Design:
> This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
> This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/kafka.py 680dd32802
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py 5b0be54982
>
>
> Diff: https://reviews.apache.org/r/60249/diff/2/
>
>
> Testing
> -------
>
> Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
>
>
> Thanks,
>
> Ambud Sharma
>
>
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Sebastian Toader <st...@hortonworks.com>.
> On June 21, 2017, 8:04 a.m., Sebastian Toader wrote:
> > Run unit ambari-server unit tests and post results here.
> >
> > Perform manual testing of upgrade scenarios eg. install a cluster that has Kafka an older version (e.g. 2.4 or 2.5.1) of Ambari that doesn't have the feature for setting rack info for Kafka. Verify that Kakfa works as expected. Than upgrade Ambari on the existing cluster to higher version that has the feature for setting rack info for Kafka and verify that Kafka still works as expected.
>
> Ambud Sharma wrote:
> Unit tests failed. If there's a specific flag I should be running them with, please let me know.
>
> Manually tested Ambari version with and without the stack.
>
>
> Tests run: 4277, Failures: 0, Errors: 0, Skipped: 32
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 11.418 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.062 s]
> [INFO] Ambari Web ......................................... SUCCESS [03:50 min]
> [INFO] Ambari Views ....................................... SUCCESS [ 3.410 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 17.426 s]
> [INFO] utility ............................................ SUCCESS [ 3.040 s]
> [INFO] ambari-metrics ..................................... SUCCESS [ 2.070 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [ 7.596 s]
> [INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 8.955 s]
> [INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 6.631 s]
> [INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [ 5.576 s]
> [INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 2.765 s]
> [INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [ 2.231 s]
> [INFO] Ambari Metrics Collector ........................... SUCCESS [05:52 min]
> [INFO] Ambari Metrics Monitor ............................. SUCCESS [ 3.240 s]
> [INFO] Ambari Metrics Grafana ............................. SUCCESS [ 33.955 s]
> [INFO] Ambari Metrics Host Aggregator ..................... SUCCESS [ 7.216 s]
> [INFO] Ambari Metrics Assembly ............................ SUCCESS [ 18.381 s]
> [INFO] Service Advisor .................................... SUCCESS [ 1.301 s]
> [INFO] Ambari Server ...................................... FAILURE [32:36 min]
> [INFO] Ambari Functional Tests ............................ SKIPPED
> [INFO] Ambari Agent ....................................... SKIPPED
> [INFO] Ambari Client ...................................... SKIPPED
> [INFO] Ambari Python Client ............................... SKIPPED
> [INFO] Ambari Groovy Client ............................... SKIPPED
> [INFO] Ambari Shell ....................................... SKIPPED
> [INFO] Ambari Python Shell ................................ SKIPPED
> [INFO] Ambari Groovy Shell ................................ SKIPPED
> [INFO] ambari-logsearch ................................... SKIPPED
> [INFO] Ambari Logsearch Appender .......................... SKIPPED
> [INFO] Ambari Logsearch Config Api ........................ SKIPPED
> [INFO] Ambari Logsearch Config Zookeeper .................. SKIPPED
> [INFO] Ambari Logsearch Web ............................... SKIPPED
> [INFO] Ambari Logsearch Server ............................ SKIPPED
> [INFO] Ambari Logsearch Log Feeder ........................ SKIPPED
> [INFO] Ambari Logsearch Assembly .......................... SKIPPED
> [INFO] Ambari Logsearch Integration Test .................. SKIPPED
> [INFO] ambari-infra ....................................... SKIPPED
> [INFO] Ambari Infra Solr Client ........................... SKIPPED
> [INFO] Ambari Infra Solr Plugin ........................... SKIPPED
> [INFO] Ambari Infra Manager ............................... SKIPPED
> [INFO] Ambari Infra Solr Assembly ......................... SKIPPED
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 44:36 min
> [INFO] Finished at: 2017-06-21T20:01:27Z
> [INFO] Final Memory: 111M/1011M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19:test (default-test) on project ambari-server: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /home/centos/ambari/ambari-server && /usr/java/jdk1.8.0_112/jre/bin/java -Xmx1024m -XX:MaxPermSize=512m -Xms512m -jar /home/centos/ambari/ambari-server/target/surefire/surefirebooter3486762047752418417.jar /home/centos/ambari/ambari-server/target/surefire/surefire7489163180494317107tmp /home/centos/ambari/ambari-server/target/surefire/surefire_4482532294765351508tmp
>
> Sebastian Toader wrote:
> Run unit tests only for ambari server: mvn clean test -am -pl ambari-server
>
> Ambud Sharma wrote:
> Failed tests:
> BufferedThreadPoolExecutorCompletionServiceTest.testMaxPoolSizeThreadsLaunched:143 expected:<10> but was:<9>
>
> Tests run: 4954, Failures: 1, Errors: 0, Skipped: 35
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 11.419 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.089 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 4.067 s]
> [INFO] utility ............................................ SUCCESS [ 3.593 s]
> [INFO] ambari-metrics ..................................... SUCCESS [ 1.401 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [ 8.732 s]
> [INFO] Service Advisor .................................... SUCCESS [ 0.728 s]
> [INFO] Ambari Server ...................................... FAILURE [48:12 min]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 48:44 min
> [INFO] Finished at: 2017-06-22T16:20:36Z
> [INFO] Final Memory: 68M/579M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19:test (default-test) on project ambari-server: There are test failures.
> [ERROR]
> [ERROR] Please refer to /home/centos/ambari/ambari-server/target/surefire-reports for the individual test results.
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the command
> [ERROR] mvn <goals> -rf :ambari-server
This unit test failure doesn't seem to be related to your change. Can you apply your patch on branch-2.5 and re-run the unit tests as well?
- Sebastian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/#review178450
-----------------------------------------------------------
On June 21, 2017, 10:33 p.m., Ambud Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60249/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 10:33 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
>
> As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
> Design:
> This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
> This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/kafka.py 680dd32802
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py 5b0be54982
>
>
> Diff: https://reviews.apache.org/r/60249/diff/2/
>
>
> Testing
> -------
>
> Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
>
>
> Thanks,
>
> Ambud Sharma
>
>
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Ambud Sharma <as...@gmail.com>.
> On June 21, 2017, 6:04 a.m., Sebastian Toader wrote:
> > Run unit ambari-server unit tests and post results here.
> >
> > Perform manual testing of upgrade scenarios eg. install a cluster that has Kafka an older version (e.g. 2.4 or 2.5.1) of Ambari that doesn't have the feature for setting rack info for Kafka. Verify that Kakfa works as expected. Than upgrade Ambari on the existing cluster to higher version that has the feature for setting rack info for Kafka and verify that Kafka still works as expected.
>
> Ambud Sharma wrote:
> Unit tests failed. If there's a specific flag I should be running them with, please let me know.
>
> Manually tested Ambari version with and without the stack.
>
>
> Tests run: 4277, Failures: 0, Errors: 0, Skipped: 32
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 11.418 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.062 s]
> [INFO] Ambari Web ......................................... SUCCESS [03:50 min]
> [INFO] Ambari Views ....................................... SUCCESS [ 3.410 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 17.426 s]
> [INFO] utility ............................................ SUCCESS [ 3.040 s]
> [INFO] ambari-metrics ..................................... SUCCESS [ 2.070 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [ 7.596 s]
> [INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 8.955 s]
> [INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 6.631 s]
> [INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [ 5.576 s]
> [INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 2.765 s]
> [INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [ 2.231 s]
> [INFO] Ambari Metrics Collector ........................... SUCCESS [05:52 min]
> [INFO] Ambari Metrics Monitor ............................. SUCCESS [ 3.240 s]
> [INFO] Ambari Metrics Grafana ............................. SUCCESS [ 33.955 s]
> [INFO] Ambari Metrics Host Aggregator ..................... SUCCESS [ 7.216 s]
> [INFO] Ambari Metrics Assembly ............................ SUCCESS [ 18.381 s]
> [INFO] Service Advisor .................................... SUCCESS [ 1.301 s]
> [INFO] Ambari Server ...................................... FAILURE [32:36 min]
> [INFO] Ambari Functional Tests ............................ SKIPPED
> [INFO] Ambari Agent ....................................... SKIPPED
> [INFO] Ambari Client ...................................... SKIPPED
> [INFO] Ambari Python Client ............................... SKIPPED
> [INFO] Ambari Groovy Client ............................... SKIPPED
> [INFO] Ambari Shell ....................................... SKIPPED
> [INFO] Ambari Python Shell ................................ SKIPPED
> [INFO] Ambari Groovy Shell ................................ SKIPPED
> [INFO] ambari-logsearch ................................... SKIPPED
> [INFO] Ambari Logsearch Appender .......................... SKIPPED
> [INFO] Ambari Logsearch Config Api ........................ SKIPPED
> [INFO] Ambari Logsearch Config Zookeeper .................. SKIPPED
> [INFO] Ambari Logsearch Web ............................... SKIPPED
> [INFO] Ambari Logsearch Server ............................ SKIPPED
> [INFO] Ambari Logsearch Log Feeder ........................ SKIPPED
> [INFO] Ambari Logsearch Assembly .......................... SKIPPED
> [INFO] Ambari Logsearch Integration Test .................. SKIPPED
> [INFO] ambari-infra ....................................... SKIPPED
> [INFO] Ambari Infra Solr Client ........................... SKIPPED
> [INFO] Ambari Infra Solr Plugin ........................... SKIPPED
> [INFO] Ambari Infra Manager ............................... SKIPPED
> [INFO] Ambari Infra Solr Assembly ......................... SKIPPED
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 44:36 min
> [INFO] Finished at: 2017-06-21T20:01:27Z
> [INFO] Final Memory: 111M/1011M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19:test (default-test) on project ambari-server: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /home/centos/ambari/ambari-server && /usr/java/jdk1.8.0_112/jre/bin/java -Xmx1024m -XX:MaxPermSize=512m -Xms512m -jar /home/centos/ambari/ambari-server/target/surefire/surefirebooter3486762047752418417.jar /home/centos/ambari/ambari-server/target/surefire/surefire7489163180494317107tmp /home/centos/ambari/ambari-server/target/surefire/surefire_4482532294765351508tmp
>
> Sebastian Toader wrote:
> Run unit tests only for ambari server: mvn clean test -am -pl ambari-server
Failed tests:
BufferedThreadPoolExecutorCompletionServiceTest.testMaxPoolSizeThreadsLaunched:143 expected:<10> but was:<9>
Tests run: 4954, Failures: 1, Errors: 0, Skipped: 35
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 11.419 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.089 s]
[INFO] Ambari Views ....................................... SUCCESS [ 4.067 s]
[INFO] utility ............................................ SUCCESS [ 3.593 s]
[INFO] ambari-metrics ..................................... SUCCESS [ 1.401 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 8.732 s]
[INFO] Service Advisor .................................... SUCCESS [ 0.728 s]
[INFO] Ambari Server ...................................... FAILURE [48:12 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 48:44 min
[INFO] Finished at: 2017-06-22T16:20:36Z
[INFO] Final Memory: 68M/579M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19:test (default-test) on project ambari-server: There are test failures.
[ERROR]
[ERROR] Please refer to /home/centos/ambari/ambari-server/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :ambari-server
> On June 21, 2017, 6:04 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
> > Line 104 (original), 104 (patched)
> > <https://reviews.apache.org/r/60249/diff/1/?file=1754688#file1754688line104>
> >
> > According to https://issues.apache.org/jira/browse/KAFKA-1215 rack awarness only supported starting from version 0.10.0.0. Shouldn't this change be implemented in common-services/KAFKA/0.10.0 instead of common-services/KAFKA/0.8.1 ?
> >
> > If setting ```broker.rack``` can cause issues for older Kafka version that doesn't support rack awarness than the setting of ```broker.rack``` has to be added to the scripts of the appropriate Kafka version on Ambari stack definition or via stack_features.
>
> Ambud Sharma wrote:
> Added a new patch that covers 0.10.0.3 common-services version of the stack. The broker.rack property if set on older version of Kafka will simply get ignored (verified with 0.8.2.2)
>
> Sebastian Toader wrote:
> Please address my other comment with regards to topology script as well. Users wanting to use their own custom logic for rack information will upload their own topology script. Currently they have to deploy themselves to all host their custom script but for Ambari 3.0.0 we're adding a feature that will deploy user provided topology script to the hosts for them.
I have commented on that below, as of Ambari 3.0.0 if that script will always be available in /etc/hadoop/ regardless of HDFS or YARN being installed then we can surely plan on how to include it. However, if that script doesn't get pushed out for Kafka install then it's difficult to enable that.
- Ambud
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/#review178450
-----------------------------------------------------------
On June 21, 2017, 8:33 p.m., Ambud Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60249/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 8:33 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
>
> As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
> Design:
> This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
> This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/kafka.py 680dd32802
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py 5b0be54982
>
>
> Diff: https://reviews.apache.org/r/60249/diff/2/
>
>
> Testing
> -------
>
> Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
>
>
> Thanks,
>
> Ambud Sharma
>
>
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Sebastian Toader <st...@hortonworks.com>.
> On June 21, 2017, 8:04 a.m., Sebastian Toader wrote:
> > Run unit ambari-server unit tests and post results here.
> >
> > Perform manual testing of upgrade scenarios eg. install a cluster that has Kafka an older version (e.g. 2.4 or 2.5.1) of Ambari that doesn't have the feature for setting rack info for Kafka. Verify that Kakfa works as expected. Than upgrade Ambari on the existing cluster to higher version that has the feature for setting rack info for Kafka and verify that Kafka still works as expected.
>
> Ambud Sharma wrote:
> Unit tests failed. If there's a specific flag I should be running them with, please let me know.
>
> Manually tested Ambari version with and without the stack.
>
>
> Tests run: 4277, Failures: 0, Errors: 0, Skipped: 32
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 11.418 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.062 s]
> [INFO] Ambari Web ......................................... SUCCESS [03:50 min]
> [INFO] Ambari Views ....................................... SUCCESS [ 3.410 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 17.426 s]
> [INFO] utility ............................................ SUCCESS [ 3.040 s]
> [INFO] ambari-metrics ..................................... SUCCESS [ 2.070 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [ 7.596 s]
> [INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 8.955 s]
> [INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 6.631 s]
> [INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [ 5.576 s]
> [INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 2.765 s]
> [INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [ 2.231 s]
> [INFO] Ambari Metrics Collector ........................... SUCCESS [05:52 min]
> [INFO] Ambari Metrics Monitor ............................. SUCCESS [ 3.240 s]
> [INFO] Ambari Metrics Grafana ............................. SUCCESS [ 33.955 s]
> [INFO] Ambari Metrics Host Aggregator ..................... SUCCESS [ 7.216 s]
> [INFO] Ambari Metrics Assembly ............................ SUCCESS [ 18.381 s]
> [INFO] Service Advisor .................................... SUCCESS [ 1.301 s]
> [INFO] Ambari Server ...................................... FAILURE [32:36 min]
> [INFO] Ambari Functional Tests ............................ SKIPPED
> [INFO] Ambari Agent ....................................... SKIPPED
> [INFO] Ambari Client ...................................... SKIPPED
> [INFO] Ambari Python Client ............................... SKIPPED
> [INFO] Ambari Groovy Client ............................... SKIPPED
> [INFO] Ambari Shell ....................................... SKIPPED
> [INFO] Ambari Python Shell ................................ SKIPPED
> [INFO] Ambari Groovy Shell ................................ SKIPPED
> [INFO] ambari-logsearch ................................... SKIPPED
> [INFO] Ambari Logsearch Appender .......................... SKIPPED
> [INFO] Ambari Logsearch Config Api ........................ SKIPPED
> [INFO] Ambari Logsearch Config Zookeeper .................. SKIPPED
> [INFO] Ambari Logsearch Web ............................... SKIPPED
> [INFO] Ambari Logsearch Server ............................ SKIPPED
> [INFO] Ambari Logsearch Log Feeder ........................ SKIPPED
> [INFO] Ambari Logsearch Assembly .......................... SKIPPED
> [INFO] Ambari Logsearch Integration Test .................. SKIPPED
> [INFO] ambari-infra ....................................... SKIPPED
> [INFO] Ambari Infra Solr Client ........................... SKIPPED
> [INFO] Ambari Infra Solr Plugin ........................... SKIPPED
> [INFO] Ambari Infra Manager ............................... SKIPPED
> [INFO] Ambari Infra Solr Assembly ......................... SKIPPED
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 44:36 min
> [INFO] Finished at: 2017-06-21T20:01:27Z
> [INFO] Final Memory: 111M/1011M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19:test (default-test) on project ambari-server: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /home/centos/ambari/ambari-server && /usr/java/jdk1.8.0_112/jre/bin/java -Xmx1024m -XX:MaxPermSize=512m -Xms512m -jar /home/centos/ambari/ambari-server/target/surefire/surefirebooter3486762047752418417.jar /home/centos/ambari/ambari-server/target/surefire/surefire7489163180494317107tmp /home/centos/ambari/ambari-server/target/surefire/surefire_4482532294765351508tmp
Run unit tests only for ambari server: mvn clean test -am -pl ambari-server
> On June 21, 2017, 8:04 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
> > Line 104 (original), 104 (patched)
> > <https://reviews.apache.org/r/60249/diff/1/?file=1754688#file1754688line104>
> >
> > According to https://issues.apache.org/jira/browse/KAFKA-1215 rack awarness only supported starting from version 0.10.0.0. Shouldn't this change be implemented in common-services/KAFKA/0.10.0 instead of common-services/KAFKA/0.8.1 ?
> >
> > If setting ```broker.rack``` can cause issues for older Kafka version that doesn't support rack awarness than the setting of ```broker.rack``` has to be added to the scripts of the appropriate Kafka version on Ambari stack definition or via stack_features.
>
> Ambud Sharma wrote:
> Added a new patch that covers 0.10.0.3 common-services version of the stack. The broker.rack property if set on older version of Kafka will simply get ignored (verified with 0.8.2.2)
Please address my other comment with regards to topology script as well. Users wanting to use their own custom logic for rack information will upload their own topology script. Currently they have to deploy themselves to all host their custom script but for Ambari 3.0.0 we're adding a feature that will deploy user provided topology script to the hosts for them.
- Sebastian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/#review178450
-----------------------------------------------------------
On June 21, 2017, 10:33 p.m., Ambud Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60249/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 10:33 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
>
> As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
> Design:
> This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
> This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/kafka.py 680dd32802
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py 5b0be54982
>
>
> Diff: https://reviews.apache.org/r/60249/diff/2/
>
>
> Testing
> -------
>
> Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
>
>
> Thanks,
>
> Ambud Sharma
>
>
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/#review178450
-----------------------------------------------------------
Fix it, then Ship it!
Run unit ambari-server unit tests and post results here.
Perform manual testing of upgrade scenarios eg. install a cluster that has Kafka an older version (e.g. 2.4 or 2.5.1) of Ambari that doesn't have the feature for setting rack info for Kafka. Verify that Kakfa works as expected. Than upgrade Ambari on the existing cluster to higher version that has the feature for setting rack info for Kafka and verify that Kafka still works as expected.
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
Line 104 (original), 104 (patched)
<https://reviews.apache.org/r/60249/#comment252447>
According to https://issues.apache.org/jira/browse/KAFKA-1215 rack awarness only supported starting from version 0.10.0.0. Shouldn't this change be implemented in common-services/KAFKA/0.10.0 instead of common-services/KAFKA/0.8.1 ?
If setting ```broker.rack``` can cause issues for older Kafka version that doesn't support rack awarness than the setting of ```broker.rack``` has to be added to the scripts of the appropriate Kafka version on Ambari stack definition or via stack_features.
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
Lines 106-115 (patched)
<https://reviews.apache.org/r/60249/#comment252445>
The ```params.all_racks``` provides only access to rack info that was set up for hosts within ambari. However there rack info may come from a custom script in that case the information in ```params.all_racks``` won't have correct data.
Use the topology script file to figure out the rack information for the current cluster. The path to the topology script is found in ```core-site/net.topology.script.file.name```
The topology script file accepts one or more host fqdn as input parameter and will return to stdout the rack info for the given hosts.
- Sebastian Toader
On June 20, 2017, 11:17 p.m., Ambud Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60249/
> -----------------------------------------------------------
>
> (Updated June 20, 2017, 11:17 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
>
> As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
> Design:
> This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
> This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py 680dd32802
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py 9acc1ef203
>
>
> Diff: https://reviews.apache.org/r/60249/diff/1/
>
>
> Testing
> -------
>
> Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
>
>
> Thanks,
>
> Ambud Sharma
>
>
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/#review178429
-----------------------------------------------------------
Fix it, then Ship it!
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
Lines 110 (patched)
<https://reviews.apache.org/r/60249/#comment252386>
Indentation looks off here, should be 2 spaces
- Alejandro Fernandez
On June 20, 2017, 9:17 p.m., Ambud Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60249/
> -----------------------------------------------------------
>
> (Updated June 20, 2017, 9:17 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
>
> As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
> Design:
> This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
> This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py 680dd32802
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py 9acc1ef203
>
>
> Diff: https://reviews.apache.org/r/60249/diff/1/
>
>
> Testing
> -------
>
> Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
>
>
> Thanks,
>
> Ambud Sharma
>
>
Re: Review Request 60249: AMBARI-21234 Ambari rack awareness for Kafka
Posted by Ambud Sharma <as...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60249/
-----------------------------------------------------------
(Updated June 20, 2017, 9:17 p.m.)
Review request for Ambari and Alejandro Fernandez.
Summary (updated)
-----------------
AMBARI-21234 Ambari rack awareness for Kafka
Repository: ambari
Description (updated)
-------
Ambari rack awareness for Kafka. https://issues.apache.org/jira/browse/AMBARI-21234
As an operations person it would be nice to manage Kafka rack awareness via Ambari. Ambari allows node rack information to be configured and this information can then be pulled in the Kafka stack and populated in the server.properties file for Kafka.
Design:
This stack change uses the /clusterHostInfo/all_hosts and /clusterHostInfo/all_racks paths and materializes them to a variable. Then it uses linear search to find this node in the list of all hosts and it's corresponding rack id. This information is then stored in a variable called rack and which is materialized during the configure method of the broker scripts.
This stack change relies on the node rack information stored in Ambari therefore will enable both Ambari UI and Blueprints to be used for setting up Kafka broker rack information.
Diffs
-----
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py 680dd32802
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py 9acc1ef203
Diff: https://reviews.apache.org/r/60249/diff/1/
Testing
-------
Manually deployed Kafka cluster and verified the broker.rack property is correctly populated.
Thanks,
Ambud Sharma