You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Jeff Zhang <zj...@apache.org> on 2016/01/08 02:18:30 UTC
Review Request 42051: [AMBARI-14561] Configuration changes of spark
thrift server for Spark 1.6
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
Review request for Ambari and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Matt <mi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review113422
-----------------------------------------------------------
Ship it!
Ship It!
- Matt
On Jan. 7, 2016, 5:18 p.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2016, 5:18 p.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115083
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py (line 161)
<https://reviews.apache.org/r/42051/#comment175943>
check for None first.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py (line 94)
<https://reviews.apache.org/r/42051/#comment175944>
format_hdp_stack_version(params.version)
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml (line 626)
<https://reviews.apache.org/r/42051/#comment175945>
Also make config changes in non-rolling-upgrade-2.4.xml
- Alejandro Fernandez
On Jan. 18, 2016, 11:58 p.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2016, 11:58 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
> On Jan. 19, 2016, 6:23 p.m., Alejandro Fernandez wrote:
> > Thanks for submitting the code review. After RU from HDP 2.3 to 2.4 passes, I'll +1
> > RU requires at least 3 hosts with NameNode HA
> > EU can be done with 1 host
RU/EU from 2.3 to 2.4 is passed
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115201
-----------------------------------------------------------
On Jan. 19, 2016, 6:50 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 6:50 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Jan. 19, 2016, 6:23 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py, line 32
> > <https://reviews.apache.org/r/42051/diff/12/?file=1201139#file1201139line32>
> >
> > This needs to accept upgrade_type.
> >
> > effective_version = params.version if upgrade_type is not None else params.hdp_stack_version
> >
> > Then format that variable if it's not None and use it for all comparisons.
After this issue is fixed, Ship It.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115201
-----------------------------------------------------------
On Jan. 19, 2016, 6:50 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 6:50 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115201
-----------------------------------------------------------
Thanks for submitting the code review. After RU from HDP 2.3 to 2.4 passes, I'll +1
RU requires at least 3 hosts with NameNode HA
EU can be done with 1 host
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py (line 31)
<https://reviews.apache.org/r/42051/#comment176099>
This needs to accept upgrade_type.
effective_version = params.version if upgrade_type is not None else params.hdp_stack_version
Then format that variable if it's not None and use it for all comparisons.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py (line 35)
<https://reviews.apache.org/r/42051/#comment176100>
This one is good.
- Alejandro Fernandez
On Jan. 19, 2016, 6:50 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 6:50 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Greg Hill <gr...@rackspace.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review116131
-----------------------------------------------------------
Seeing as one of our upcoming use-cases for Ambari is just Spark + Kafka without HDFS, this change will make that impossible for us. Can we rethink this to avoid the direct HDFS dependency in Spark?
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py (line 89)
<https://reviews.apache.org/r/42051/#comment177122>
Can you please make this configurable? That is a fine default value, but the user may want to override it with a different hdfs path or a local filesystem path.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py (line 41)
<https://reviews.apache.org/r/42051/#comment177125>
Is HDFS guaranteed to exist? What if someone wanted to spin up Spark + Kafka without HDFS? We shouldn't hardcode dependencies like this, IMO.
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml (line 37)
<https://reviews.apache.org/r/42051/#comment177126>
Related to my other comments, I think with some minor tweaks, you shouldn't need to require HDFS for this.
- Greg Hill
On Jan. 19, 2016, 11:41 p.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 11:41 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115298
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Jan. 19, 2016, 11:41 p.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 11:41 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 11:41 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 11:40 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 11:23 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Changes
-------
Fix the unit test failure
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115272
-----------------------------------------------------------
Will ship it afer unit tests pass. Thanks, Alejandro
- Alejandro Fernandez
On Jan. 19, 2016, 9:02 p.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 9:02 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py d4b8f85
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 6:50 a.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 1:43 a.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 1:27 a.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 943664a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py d2145e2
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115086
-----------------------------------------------------------
LGTM, +1. Once we get an ack from Alejandro we can commit it.
- Sumit Mohanty
On Jan. 19, 2016, 12:16 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 12:16 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115089
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py (line 94)
<https://reviews.apache.org/r/42051/#comment175948>
effective_version = params.version if params.upgrade_direction in ["UPGRADE", "DOWNGRADE"] else params.hdp_stack_version
# Note, that can also use upgrade_type. If none, then use hdp_stack_version, otherwise, use params.version
params.version is only available during RU/EU. If not in that scenario, use hdp_stack_version
compare_version(format_hdp_stack_version(effective_version), "2.4.0.0") >= 0
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py (line 34)
<https://reviews.apache.org/r/42051/#comment175949>
Similar comment as above. spark_service.py and setup_spark.py should accept upgrade_type or lookup params.upgrade_direction
- Alejandro Fernandez
On Jan. 19, 2016, 12:16 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2016, 12:16 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 19, 2016, 12:16 a.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 18, 2016, 11:58 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 18, 2016, 11:50 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 34423f9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml b8bd1ee
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115077
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py (line 34)
<https://reviews.apache.org/r/42051/#comment175933>
Assuming upgrade from 2.3 to 2.4:
EU: stop on 2.3, changes desired version to 2.4, restarts on 2.4. Downgrade: hdp_stack_version is still 2.4
RU: rolling restart of all components. change desired version to 2.4 at the beginning. Downgrade: hdp_stack_version is 2.3
Instead, use params.version (e.g, 2.4.0.0-1234), which is the version either upgrading/downgrading to. There's a function to format it in order to remove the build number.
compare_versions(format_hdp_stack_version(params.version), '2.4.0.0')
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml (line 110)
<https://reviews.apache.org/r/42051/#comment175934>
Rebase
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml
<https://reviews.apache.org/r/42051/#comment175935>
No change to this file.
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml (line 106)
<https://reviews.apache.org/r/42051/#comment175936>
spark-defaults?
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml (line 118)
<https://reviews.apache.org/r/42051/#comment175937>
spark-defaults?
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml (line 671)
<https://reviews.apache.org/r/42051/#comment175938>
Wrong file, should use 2.3->2.4 for nonrolling-upgrade-2.4.xml and upgrade-2.4.xml
- Alejandro Fernandez
On Jan. 18, 2016, 11:27 p.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2016, 11:27 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 34423f9
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml b8bd1ee
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 18, 2016, 11:27 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml 34423f9
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml b8bd1ee
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 18, 2016, 10:25 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml ba2578b
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml ab91379
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 5534e3e
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 863e0a0
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml e8f1db4
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review115052
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py (line 50)
<https://reviews.apache.org/r/42051/#comment175899>
Where is this version used? Keep in mind that params are loaded at the beginning, so if any logic then calls hdp-select, then this may be the incorrect value.
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml (line 22)
<https://reviews.apache.org/r/42051/#comment175902>
This will need a separate Jira to track the config changes in HDP 2.3 to 2.4
- Alejandro Fernandez
On Jan. 16, 2016, 1:13 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2016, 1:13 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
> On Jan. 18, 2016, 1:58 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml, line 25
> > <https://reviews.apache.org/r/42051/diff/4/?file=1199119#file1199119line25>
> >
> > Existing installations will still have these properties - is that OK?
Yes, just delete some properties and add some new properties
> On Jan. 18, 2016, 1:58 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py, line 50
> > <https://reviews.apache.org/r/42051/diff/4/?file=1199114#file1199114line50>
> >
> > When did spark get added. Will get_hdp_version work for all versions of HDP?
Spark is added from HDP 2.2, this might be an issue, will fix it.
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review114925
-----------------------------------------------------------
On Jan. 16, 2016, 1:13 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2016, 1:13 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
> On Jan. 18, 2016, 1:58 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml, line 70
> > <https://reviews.apache.org/r/42051/diff/4/?file=1199119#file1199119line70>
> >
> > Ambari upgrade will automatically add these properties for already installed cluster. Is that OK? can you test one upgrade scenario where spark is already installed using an older version of Ambari?
How can I do an ambari upgrade with this change ?
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review114925
-----------------------------------------------------------
On Jan. 16, 2016, 1:13 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2016, 1:13 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Sumit Mohanty <sm...@hortonworks.com>.
> On Jan. 18, 2016, 1:58 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml, line 70
> > <https://reviews.apache.org/r/42051/diff/4/?file=1199119#file1199119line70>
> >
> > Ambari upgrade will automatically add these properties for already installed cluster. Is that OK? can you test one upgrade scenario where spark is already installed using an older version of Ambari?
>
> Jeff Zhang wrote:
> How can I do an ambari upgrade with this change ?
This is a non-issue.
- Sumit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review114925
-----------------------------------------------------------
On Jan. 16, 2016, 1:13 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2016, 1:13 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review114925
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py (line 50)
<https://reviews.apache.org/r/42051/#comment175757>
When did spark get added. Will get_hdp_version work for all versions of HDP?
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py (line 37)
<https://reviews.apache.org/r/42051/#comment175758>
This may have RU implications. Can you discuss with Alejandro/Nate?
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml (line 25)
<https://reviews.apache.org/r/42051/#comment175759>
Existing installations will still have these properties - is that OK?
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml (line 70)
<https://reviews.apache.org/r/42051/#comment175760>
Ambari upgrade will automatically add these properties for already installed cluster. Is that OK? can you test one upgrade scenario where spark is already installed using an older version of Ambari?
- Sumit Mohanty
On Jan. 16, 2016, 1:13 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2016, 1:13 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 16, 2016, 1:13 a.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py d4c6732
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 15, 2016, 7:26 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------
(Updated Jan. 15, 2016, 6:51 p.m.)
Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.
Bugs: AMBARI-14561
https://issues.apache.org/jira/browse/AMBARI-14561
Repository: ambari
Description
-------
This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 68240bd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
Diff: https://reviews.apache.org/r/42051/diff/
Testing
-------
This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
Thanks,
Jeff Zhang
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Sumit Mohanty <sm...@hortonworks.com>.
> On Jan. 8, 2016, 6:06 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2, line 23
> > <https://reviews.apache.org/r/42051/diff/1/?file=1186627#file1186627line23>
> >
> > There is no need to make these configurable or editable?
>
> Jeff Zhang wrote:
> It would be better if it could.
Then you can model this as config files with content field (e.g. atlas-env.xml).
- Sumit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review113417
-----------------------------------------------------------
On Jan. 8, 2016, 1:18 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:18 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
> On Jan. 8, 2016, 6:06 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2, line 23
> > <https://reviews.apache.org/r/42051/diff/1/?file=1186627#file1186627line23>
> >
> > There is no need to make these configurable or editable?
It would be better if it could.
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review113417
-----------------------------------------------------------
On Jan. 8, 2016, 1:18 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:18 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review113417
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 (line 23)
<https://reviews.apache.org/r/42051/#comment174046>
There is no need to make these configurable or editable?
- Sumit Mohanty
On Jan. 8, 2016, 1:18 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:18 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Jeff Zhang <zj...@apache.org>.
> On Jan. 8, 2016, 6:07 a.m., Sumit Mohanty wrote:
> > Ship It!
Could you please hold on ? Actually I havn't resolved the follow thing, could you help on that ?
The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. What I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review113419
-----------------------------------------------------------
On Jan. 8, 2016, 1:18 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:18 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>
Re: Review Request 42051: [AMBARI-14561] Configuration changes of
spark thrift server for Spark 1.6
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/#review113419
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Jan. 8, 2016, 1:18 a.m., Jeff Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42051/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:18 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Bugs: AMBARI-14561
> https://issues.apache.org/jira/browse/AMBARI-14561
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
> - Remove advanced properties except queue name
> - Use yarn-client mode
> - Enable Dynamic Resource Allocation
> - Enable Fair scheduling policy
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py 4b38572
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/templates/spark-thrift-fairscheduler.xml.j2 PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-hive-site-override.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/quicklinks/quicklinks.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/42051/diff/
>
>
> Testing
> -------
>
> This patch doesn't solve the issue completly. The weird thing is that the configuration in HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml. And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP 2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml in HDP 2.3
>
>
> Thanks,
>
> Jeff Zhang
>
>