You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Gautam Borad <gb...@gmail.com> on 2015/02/06 17:38:44 UTC
Review Request 30733: AMBARI-9494 : Support Spark installation via
Ambari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/
-----------------------------------------------------------
Review request for Ambari, Jaimin Jetly and Yusaku Sako.
Bugs: AMBARI-9494
https://issues.apache.org/jira/browse/AMBARI-9494
Repository: ambari
Description
-------
Add support for management for Spark using Ambari.
Support for following components are to be addressed :
1] Spark Client
2] Spark Job History Server
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION
Diff: https://reviews.apache.org/r/30733/diff/
Testing
-------
Did the following testing:
1] Tested on RHEL6 single node cluster
2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client on other nodes
3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark History Server
Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0, even if the HDP version is 2.2.1.0-xxxx
Thanks,
Gautam Borad
Re: Review Request 30733: AMBARI-9494 : Support Spark installation
via Ambari
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/#review71749
-----------------------------------------------------------
Pushed to trunk in commit 43352a755adff5db84ed8c8cfc8c5d6fa60eff82
- Alejandro Fernandez
On Feb. 9, 2015, 1:32 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30733/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 1:32 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly and Yusaku Sako.
>
>
> Bugs: AMBARI-9494
> https://issues.apache.org/jira/browse/AMBARI-9494
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Add support for management for Spark using Ambari.
> Support for following components are to be addressed :
> 1] Spark Client
> 2] Spark Job History Server
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/alerts.json PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30733/diff/
>
>
> Testing
> -------
>
> Did the following testing:
>
> 1] Tested on RHEL6 single node cluster
> 2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client on other nodes
> 3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark History Server
>
> Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0, even if the HDP version is 2.2.1.0-xxxx
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 30733: AMBARI-9494 : Support Spark installation
via Ambari
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/#review71746
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Feb. 9, 2015, 1:32 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30733/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 1:32 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly and Yusaku Sako.
>
>
> Bugs: AMBARI-9494
> https://issues.apache.org/jira/browse/AMBARI-9494
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Add support for management for Spark using Ambari.
> Support for following components are to be addressed :
> 1] Spark Client
> 2] Spark Job History Server
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/alerts.json PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30733/diff/
>
>
> Testing
> -------
>
> Did the following testing:
>
> 1] Tested on RHEL6 single node cluster
> 2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client on other nodes
> 3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark History Server
>
> Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0, even if the HDP version is 2.2.1.0-xxxx
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 30733: AMBARI-9494 : Support Spark installation
via Ambari
Posted by Gautam Borad <gb...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/
-----------------------------------------------------------
(Updated Feb. 9, 2015, 1:32 p.m.)
Review request for Ambari, Jaimin Jetly and Yusaku Sako.
Changes
-------
Added AMBARI-9494-Support-Spark-installation-via-Ambari-ga.2.patch which handles all fixes ( except the named imports )
Bugs: AMBARI-9494
https://issues.apache.org/jira/browse/AMBARI-9494
Repository: ambari
Description
-------
Add support for management for Spark using Ambari.
Support for following components are to be addressed :
1] Spark Client
2] Spark Job History Server
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/alerts.json PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py PRE-CREATION
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION
Diff: https://reviews.apache.org/r/30733/diff/
Testing
-------
Did the following testing:
1] Tested on RHEL6 single node cluster
2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client on other nodes
3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark History Server
Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0, even if the HDP version is 2.2.1.0-xxxx
Thanks,
Gautam Borad
Re: Review Request 30733: AMBARI-9494 : Support Spark installation
via Ambari
Posted by Gautam Borad <gb...@gmail.com>.
> On Feb. 6, 2015, 6:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml, line 69
> > <https://reviews.apache.org/r/30733/diff/1/?file=852169#file852169line69>
> >
> > May as well remove this property.
Will fix.
> On Feb. 6, 2015, 6:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py, line 32
> > <https://reviews.apache.org/r/30733/diff/1/?file=852174#file852174line32>
> >
> > From RU perspective, this looks good. We will need hdp-select to support the "spark" component.
hdp-select supports 'spark-client' and 'spark-historyserver' currently.
> On Feb. 6, 2015, 6:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py, line 63
> > <https://reviews.apache.org/r/30733/diff/1/?file=852174#file852174line63>
> >
> > Remove all of the unnecessary "pass" calls
Will fix.
- Gautam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/#review71464
-----------------------------------------------------------
On Feb. 9, 2015, 1:32 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30733/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 1:32 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly and Yusaku Sako.
>
>
> Bugs: AMBARI-9494
> https://issues.apache.org/jira/browse/AMBARI-9494
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Add support for management for Spark using Ambari.
> Support for following components are to be addressed :
> 1] Spark Client
> 2] Spark Job History Server
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/alerts.json PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30733/diff/
>
>
> Testing
> -------
>
> Did the following testing:
>
> 1] Tested on RHEL6 single node cluster
> 2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client on other nodes
> 3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark History Server
>
> Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0, even if the HDP version is 2.2.1.0-xxxx
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 30733: AMBARI-9494 : Support Spark installation
via Ambari
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/#review71464
-----------------------------------------------------------
Ship it!
I'm ok with this going in now, and fixing the named imports laters.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml
<https://reviews.apache.org/r/30733/#comment117197>
May as well remove this property.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/30733/#comment117199>
Preferable to use named imports instead of *
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/30733/#comment117200>
From RU perspective, this looks good. We will need hdp-select to support the "spark" component.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/30733/#comment117203>
Remove all of the unnecessary "pass" calls
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
<https://reviews.apache.org/r/30733/#comment117206>
Will need to consider this when switching between versions.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py
<https://reviews.apache.org/r/30733/#comment117204>
Use named imports.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py
<https://reviews.apache.org/r/30733/#comment117208>
This may need to be generalized depending on the stack type, e.g., HDP.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py
<https://reviews.apache.org/r/30733/#comment117209>
Should this use hadoop-client instead?
- Alejandro Fernandez
On Feb. 6, 2015, 4:38 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30733/
> -----------------------------------------------------------
>
> (Updated Feb. 6, 2015, 4:38 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly and Yusaku Sako.
>
>
> Bugs: AMBARI-9494
> https://issues.apache.org/jira/browse/AMBARI-9494
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Add support for management for Spark using Ambari.
> Support for following components are to be addressed :
> 1] Spark Client
> 2] Spark Job History Server
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py PRE-CREATION
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30733/diff/
>
>
> Testing
> -------
>
> Did the following testing:
>
> 1] Tested on RHEL6 single node cluster
> 2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client on other nodes
> 3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark History Server
>
> Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0, even if the HDP version is 2.2.1.0-xxxx
>
>
> Thanks,
>
> Gautam Borad
>
>