You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/05/23 02:54:00 UTC
Re: Review Request 34618: Spark History Server in HDP 2.3 no longer
needs
tez.tar.gz, which now needs to be uploaded to HDFS during Tez Service Check
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34618/
-----------------------------------------------------------
(Updated May 23, 2015, 12:53 a.m.)
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Bugs: AMBARI-11357
https://issues.apache.org/jira/browse/AMBARI-11357
Repository: ambari
Description
-------
In HDP 2.3, Spark no longer needs Tez as a dependency.
This means that
1. Spark metainfo.xml does not need to co-host Tez-client
2. Spark History Server restart or RU restart does not need to copy tez tarball to HDFS. Note, this is still needed in HDP 2.2
3. Tez needs to be responsible for copying its own tarball to HDFS because its service check depends on it.
Diffs
-----
ambari-agent/conf/windows/service_wrapper.py 0851745
ambari-common/src/main/python/ambari_commons/os_windows.py 3df44af
ambari-common/src/main/python/resource_management/core/providers/package/choco.py 531372c
ambari-common/src/main/python/resource_management/core/providers/windows/service.py 8e6ddff
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py 507336b
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py 9260683
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py e5ee7f9
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_mapping.py 6a96c28
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/status.py 91f13d4
Diff: https://reviews.apache.org/r/34618/diff/
Testing (updated)
-------
----------------------------------------------------------------------
Total run:742
Total errors:0
Total failures:0
OK
Copied changes files to a cluster with HDP 2.3.0.0 and HDFS, YARN, MR, ZK.
Then installed Spark, which did not require Tez, as expected.
Then installed Tez, whose service checked passed.
Then ran an RU and everything passed
Thanks,
Alejandro Fernandez
Re: Review Request 34618: Spark History Server in HDP 2.3 no longer
needs
tez.tar.gz, which now needs to be uploaded to HDFS during Tez Service Check
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34618/
-----------------------------------------------------------
(Updated May 25, 2015, 8:31 a.m.)
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Changes
-------
Addressed comments, ready to commit.
Bugs: AMBARI-11357
https://issues.apache.org/jira/browse/AMBARI-11357
Repository: ambari
Description
-------
In HDP 2.3, Spark no longer needs Tez as a dependency.
This means that
1. Spark metainfo.xml does not need to co-host Tez-client
2. Spark History Server restart or RU restart does not need to copy tez tarball to HDFS. Note, this is still needed in HDP 2.2
3. Tez needs to be responsible for copying its own tarball to HDFS because its service check depends on it.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 106b8cc
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py 834e208
ambari-server/src/main/resources/common-services/SPARK/1.3.1.2.3/metainfo.xml PRE-CREATION
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py 98973ae
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 205ccc9
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py dbefa5c
Diff: https://reviews.apache.org/r/34618/diff/
Testing
-------
----------------------------------------------------------------------
Total run:742
Total errors:0
Total failures:0
OK
Copied changes files to a cluster with HDP 2.3.0.0 and HDFS, YARN, MR, ZK.
Then installed Spark, which did not require Tez, as expected.
Then installed Tez, whose service checked passed.
Then ran an RU and everything passed
Thanks,
Alejandro Fernandez
Re: Review Request 34618: Spark History Server in HDP 2.3 no longer
needs
tez.tar.gz, which now needs to be uploaded to HDFS during Tez Service Check
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34618/#review85051
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py
<https://reviews.apache.org/r/34618/#comment136522>
In HDP 2.2, it would rely on other services to copy the tarball to HDFS, primarily Hive, Pig, and then Spark in 2.2.4.2. Technically, it should have been responsible too, so I'll fix this.
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml
<https://reviews.apache.org/r/34618/#comment136523>
I can create a SPARK folder in common-services with the newer version of Spark, and remove TEZ from that metainfo.xml file, so that this file in HDP 2.3 would not need anything.
- Alejandro Fernandez
On May 23, 2015, 1:52 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34618/
> -----------------------------------------------------------
>
> (Updated May 23, 2015, 1:52 a.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-11357
> https://issues.apache.org/jira/browse/AMBARI-11357
>
>
> Repository: ambari
>
>
> Description
> -------
>
> In HDP 2.3, Spark no longer needs Tez as a dependency.
> This means that
> 1. Spark metainfo.xml does not need to co-host Tez-client
> 2. Spark History Server restart or RU restart does not need to copy tez tarball to HDFS. Note, this is still needed in HDP 2.2
> 3. Tez needs to be responsible for copying its own tarball to HDFS because its service check depends on it.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 106b8cc
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py 834e208
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py 98973ae
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 205ccc9
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 489037c
>
> Diff: https://reviews.apache.org/r/34618/diff/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Total run:742
> Total errors:0
> Total failures:0
> OK
>
> Copied changes files to a cluster with HDP 2.3.0.0 and HDFS, YARN, MR, ZK.
> Then installed Spark, which did not require Tez, as expected.
> Then installed Tez, whose service checked passed.
> Then ran an RU and everything passed
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34618: Spark History Server in HDP 2.3 no longer
needs
tez.tar.gz, which now needs to be uploaded to HDFS during Tez Service Check
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34618/#review85049
-----------------------------------------------------------
Ship it!
Some comments, mostly on my lack of knowledge on Spark and Tez.
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py
<https://reviews.apache.org/r/34618/#comment136518>
Thank you!
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py
<https://reviews.apache.org/r/34618/#comment136519>
I just want to make sure we have the >= correct here. Tez needs to copy its own tarball to HDFS ... but that's not restricted to 2.3+ ... it's going to do it for 2.2 as well, right?
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml
<https://reviews.apache.org/r/34618/#comment136520>
Does Spark need Tez removed from its metainfo? If that's what this code is doing, then we're OK, but at first glace, it seems like the dependency is still there.
If I'm wrong, then you can drop the issue.
- Jonathan Hurley
On May 22, 2015, 9:52 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34618/
> -----------------------------------------------------------
>
> (Updated May 22, 2015, 9:52 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-11357
> https://issues.apache.org/jira/browse/AMBARI-11357
>
>
> Repository: ambari
>
>
> Description
> -------
>
> In HDP 2.3, Spark no longer needs Tez as a dependency.
> This means that
> 1. Spark metainfo.xml does not need to co-host Tez-client
> 2. Spark History Server restart or RU restart does not need to copy tez tarball to HDFS. Note, this is still needed in HDP 2.2
> 3. Tez needs to be responsible for copying its own tarball to HDFS because its service check depends on it.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 106b8cc
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py 834e208
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py 98973ae
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 205ccc9
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 489037c
>
> Diff: https://reviews.apache.org/r/34618/diff/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Total run:742
> Total errors:0
> Total failures:0
> OK
>
> Copied changes files to a cluster with HDP 2.3.0.0 and HDFS, YARN, MR, ZK.
> Then installed Spark, which did not require Tez, as expected.
> Then installed Tez, whose service checked passed.
> Then ran an RU and everything passed
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34618: Spark History Server in HDP 2.3 no longer
needs
tez.tar.gz, which now needs to be uploaded to HDFS during Tez Service Check
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34618/
-----------------------------------------------------------
(Updated May 23, 2015, 1:52 a.m.)
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Changes
-------
Correct patch this time
Bugs: AMBARI-11357
https://issues.apache.org/jira/browse/AMBARI-11357
Repository: ambari
Description
-------
In HDP 2.3, Spark no longer needs Tez as a dependency.
This means that
1. Spark metainfo.xml does not need to co-host Tez-client
2. Spark History Server restart or RU restart does not need to copy tez tarball to HDFS. Note, this is still needed in HDP 2.2
3. Tez needs to be responsible for copying its own tarball to HDFS because its service check depends on it.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 106b8cc
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py 834e208
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py 98973ae
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 205ccc9
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 489037c
Diff: https://reviews.apache.org/r/34618/diff/
Testing
-------
----------------------------------------------------------------------
Total run:742
Total errors:0
Total failures:0
OK
Copied changes files to a cluster with HDP 2.3.0.0 and HDFS, YARN, MR, ZK.
Then installed Spark, which did not require Tez, as expected.
Then installed Tez, whose service checked passed.
Then ran an RU and everything passed
Thanks,
Alejandro Fernandez