You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/03/07 00:19:08 UTC
Review Request 31815: Spark on tez apps fails needs tez.tar.gz copied
to HDFS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/
-----------------------------------------------------------
Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth Gunturi, and Sid Wagle.
Bugs: AMBARI-9954
https://issues.apache.org/jira/browse/AMBARI-9954
Repository: ambari
Description
-------
The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
Currently, only Pig Service Check and Hive START copy it to HDFS.
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml ce8ad7a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 518352f
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py PRE-CREATION
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json PRE-CREATION
Diff: https://reviews.apache.org/r/31815/diff/
Testing
-------
I added a unit test,
----------------------------------------------------------------------
Total run:610
Total errors:0
Total failures:0
OK
I installed Spark and Tez at the same time, and put the Spark History Server on host 1, and Tez Client on host 2. The installation for Spark History Server has a dependency on Tez Client, so it also installed it on host 1. The Spark History Server START command then copied the tarball to /hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling Upgrade restart.
```
[root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
Found 1 items
-r--r--r-- 3 hdfs hadoop 40656789 2015-03-06 02:58 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
[root@c6408 ~]# su - spark
[spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi --master execution-context:org.apache.spark.tez.TezJobExecutionContext /usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar 3
[spark@c6408 ~]$ echo $?
0
```
Thanks,
Alejandro Fernandez
Re: Review Request 31815: Spark on tez apps fails needs tez.tar.gz
copied to HDFS
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/
-----------------------------------------------------------
(Updated March 9, 2015, 6:23 p.m.)
Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth Gunturi, and Sid Wagle.
Changes
-------
Removed import * and minimized spark-job-history-server.json
Bugs: AMBARI-9954
https://issues.apache.org/jira/browse/AMBARI-9954
Repository: ambari
Description
-------
The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
Currently, only Pig Service Check and Hive START copy it to HDFS.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml ce8ad7a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 518352f
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py PRE-CREATION
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json PRE-CREATION
Diff: https://reviews.apache.org/r/31815/diff/
Testing
-------
I added a unit test,
----------------------------------------------------------------------
Total run:610
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.420s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.032s]
[INFO] Ambari Web ........................................ SUCCESS [41.221s]
[INFO] Ambari Views ...................................... SUCCESS [2.345s]
[INFO] Ambari Admin View ................................. SUCCESS [10.019s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.446s]
[INFO] Ambari Server ..................................... SUCCESS [33:24.375s]
[INFO] Ambari Agent ...................................... SUCCESS [16.789s]
[INFO] Ambari Client ..................................... SUCCESS [0.038s]
[INFO] Ambari Python Client .............................. SUCCESS [0.257s]
[INFO] Ambari Groovy Client .............................. SUCCESS [9.922s]
[INFO] Ambari Shell ...................................... SUCCESS [0.033s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.036s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [6.632s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35:01.956s
[INFO] Finished at: Fri Mar 06 16:16:31 PST 2015
[INFO] Final Memory: 69M/565M
[INFO] ------------------------------------------------------------------------
I installed Spark and Tez at the same time, and put the Spark History Server on host 1, and Tez Client on host 2. The installation for Spark History Server has a dependency on Tez Client, so it also installed it on host 1. The Spark History Server START command then copied the tarball to /hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling Upgrade restart.
```
[root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
Found 1 items
-r--r--r-- 3 hdfs hadoop 40656789 2015-03-06 02:58 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
[root@c6408 ~]# su - spark
[spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi --master execution-context:org.apache.spark.tez.TezJobExecutionContext /usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar 3
[spark@c6408 ~]$ echo $?
0
```
Thanks,
Alejandro Fernandez
Re: Review Request 31815: Spark on tez apps fails needs tez.tar.gz
copied to HDFS
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/#review75716
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/31815/#comment122939>
Correct * import.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/31815/#comment122940>
Maybe make `tez` and `spark-historyserver` constants since they are reused down below.
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json
<https://reviews.apache.org/r/31815/#comment122941>
I try to strip out all of the unnecessary JSON when creating test resources so that they are easier to read and require less handling if something about the JSON changes in the future.
- Jonathan Hurley
On March 6, 2015, 7:29 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31815/
> -----------------------------------------------------------
>
> (Updated March 6, 2015, 7:29 p.m.)
>
>
> Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth Gunturi, and Sid Wagle.
>
>
> Bugs: AMBARI-9954
> https://issues.apache.org/jira/browse/AMBARI-9954
>
>
> Repository: ambari
>
>
> Description
> -------
>
> The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
> Currently, only Pig Service Check and Hive START copy it to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml ce8ad7a
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 518352f
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py PRE-CREATION
> ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/31815/diff/
>
>
> Testing
> -------
>
> I added a unit test,
> ----------------------------------------------------------------------
> Total run:610
> Total errors:0
> Total failures:0
> OK
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.420s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.032s]
> [INFO] Ambari Web ........................................ SUCCESS [41.221s]
> [INFO] Ambari Views ...................................... SUCCESS [2.345s]
> [INFO] Ambari Admin View ................................. SUCCESS [10.019s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.446s]
> [INFO] Ambari Server ..................................... SUCCESS [33:24.375s]
> [INFO] Ambari Agent ...................................... SUCCESS [16.789s]
> [INFO] Ambari Client ..................................... SUCCESS [0.038s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.257s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [9.922s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.033s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.036s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [6.632s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 35:01.956s
> [INFO] Finished at: Fri Mar 06 16:16:31 PST 2015
> [INFO] Final Memory: 69M/565M
> [INFO] ------------------------------------------------------------------------
>
>
> I installed Spark and Tez at the same time, and put the Spark History Server on host 1, and Tez Client on host 2. The installation for Spark History Server has a dependency on Tez Client, so it also installed it on host 1. The Spark History Server START command then copied the tarball to /hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling Upgrade restart.
>
> ```
> [root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
> Found 1 items
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-06 02:58 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
> [root@c6408 ~]# su - spark
> [spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi --master execution-context:org.apache.spark.tez.TezJobExecutionContext /usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar 3
>
> [spark@c6408 ~]$ echo $?
> 0
> ```
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31815: Spark on tez apps fails needs tez.tar.gz
copied to HDFS
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/
-----------------------------------------------------------
(Updated March 7, 2015, 12:29 a.m.)
Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth Gunturi, and Sid Wagle.
Bugs: AMBARI-9954
https://issues.apache.org/jira/browse/AMBARI-9954
Repository: ambari
Description
-------
The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
Currently, only Pig Service Check and Hive START copy it to HDFS.
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml ce8ad7a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py 518352f
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py PRE-CREATION
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json PRE-CREATION
Diff: https://reviews.apache.org/r/31815/diff/
Testing (updated)
-------
I added a unit test,
----------------------------------------------------------------------
Total run:610
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.420s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.032s]
[INFO] Ambari Web ........................................ SUCCESS [41.221s]
[INFO] Ambari Views ...................................... SUCCESS [2.345s]
[INFO] Ambari Admin View ................................. SUCCESS [10.019s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.446s]
[INFO] Ambari Server ..................................... SUCCESS [33:24.375s]
[INFO] Ambari Agent ...................................... SUCCESS [16.789s]
[INFO] Ambari Client ..................................... SUCCESS [0.038s]
[INFO] Ambari Python Client .............................. SUCCESS [0.257s]
[INFO] Ambari Groovy Client .............................. SUCCESS [9.922s]
[INFO] Ambari Shell ...................................... SUCCESS [0.033s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.036s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [6.632s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35:01.956s
[INFO] Finished at: Fri Mar 06 16:16:31 PST 2015
[INFO] Final Memory: 69M/565M
[INFO] ------------------------------------------------------------------------
I installed Spark and Tez at the same time, and put the Spark History Server on host 1, and Tez Client on host 2. The installation for Spark History Server has a dependency on Tez Client, so it also installed it on host 1. The Spark History Server START command then copied the tarball to /hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling Upgrade restart.
```
[root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
Found 1 items
-r--r--r-- 3 hdfs hadoop 40656789 2015-03-06 02:58 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
[root@c6408 ~]# su - spark
[spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi --master execution-context:org.apache.spark.tez.TezJobExecutionContext /usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar 3
[spark@c6408 ~]$ echo $?
0
```
Thanks,
Alejandro Fernandez