You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/02/11 23:42:06 UTC
Review Request 30899: Ambari does not upload Tez tarball to HDFS when
Tez is installed
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
Bugs: AMBARI-9585
https://issues.apache.org/jira/browse/AMBARI-9585
Repository: ambari
Description
-------
Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
Instead, this should happen whenever Tez is installed.
For a RU, the install of new bits should also copy the tarball to HDFS.
Diffs
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
Diff: https://reviews.apache.org/r/30899/diff/
Testing
-------
Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
Then ensured that the tez tarball was copied to hdfs,
su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
Unit tests are in progress.
Thanks,
Alejandro Fernandez
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Hitesh Shah <hi...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72090
-----------------------------------------------------------
Ship it!
The changes seems fine in that it looks like it will fix the issue that was raised. However, there seem to be other problems with the existing code that can/should be looked at in a follow-up jira.
- Hitesh Shah
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Hitesh Shah <hi...@hortonworks.com>.
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py, line 57
> > <https://reviews.apache.org/r/30899/diff/1/?file=861153#file861153line57>
> >
> > When we copy a tarball to HDFS, we need to figure out the destination, e.g., /hdp/apps/2.2.1.0-2260/tez
> >
> > That version number has to come from somewhere, so we use hdp-select status {component_name}, where component name is the 2nd argument.
>
> Hitesh Shah wrote:
> In that case, why is the version of mapreduce dependent on hive-server2?
Also, I am not sure why the mapreduce tarball is being uploaded as user tez if that is what the 3rd param is used for.
- Hitesh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72057
-----------------------------------------------------------
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py, line 57
> > <https://reviews.apache.org/r/30899/diff/1/?file=861153#file861153line57>
> >
> > When we copy a tarball to HDFS, we need to figure out the destination, e.g., /hdp/apps/2.2.1.0-2260/tez
> >
> > That version number has to come from somewhere, so we use hdp-select status {component_name}, where component name is the 2nd argument.
>
> Hitesh Shah wrote:
> In that case, why is the version of mapreduce dependent on hive-server2?
>
> Hitesh Shah wrote:
> Also, I am not sure why the mapreduce tarball is being uploaded as user tez if that is what the 3rd param is used for.
It used to be dependent on the version that hive-server2 was on. That is no longer the case, not it only depends on whatever "hdp-select status hadoop-client" returns
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py, line 38
> > <https://reviews.apache.org/r/30899/diff/1/?file=861155#file861155line38>
> >
> > When hadoop-client is repointed, that means that tez has been upaded as well. This is important during a rolling upgrade, so that we copy the tez tarball to the appropriate folder in HDFS.
>
> Hitesh Shah wrote:
> Why not just rely on the version of Tez itself? Why does it need to based on hadoop client. Please run this by the folks who are driving rolling upgrades as I am not sure of the full context.
This is because tez does not have a component in hdp-select. Instead, it is updated along with the rest of the clients, all of which rely on "hdp-select status hadoop-client".
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72057
-----------------------------------------------------------
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py, line 59
> > <https://reviews.apache.org/r/30899/diff/1/?file=861153#file861153line59>
> >
> > Will clean
Actually, my patch didn't show this as a diff, so it must be Git or Review Board showing a diff in the line ending.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72057
-----------------------------------------------------------
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Hitesh Shah <hi...@hortonworks.com>.
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py, line 57
> > <https://reviews.apache.org/r/30899/diff/1/?file=861153#file861153line57>
> >
> > When we copy a tarball to HDFS, we need to figure out the destination, e.g., /hdp/apps/2.2.1.0-2260/tez
> >
> > That version number has to come from somewhere, so we use hdp-select status {component_name}, where component name is the 2nd argument.
In that case, why is the version of mapreduce dependent on hive-server2?
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py, line 38
> > <https://reviews.apache.org/r/30899/diff/1/?file=861155#file861155line38>
> >
> > When hadoop-client is repointed, that means that tez has been upaded as well. This is important during a rolling upgrade, so that we copy the tez tarball to the appropriate folder in HDFS.
Why not just rely on the version of Tez itself? Why does it need to based on hadoop client. Please run this by the folks who are driving rolling upgrades as I am not sure of the full context.
> On Feb. 11, 2015, 11:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py, line 59
> > <https://reviews.apache.org/r/30899/diff/1/?file=861153#file861153line59>
> >
> > Will clean
>
> Alejandro Fernandez wrote:
> Actually, my patch didn't show this as a diff, so it must be Git or Review Board showing a diff in the line ending.
Correct - existing whitespace issue from a previous commit.
- Hitesh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72057
-----------------------------------------------------------
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72057
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
<https://reviews.apache.org/r/30899/#comment117990>
When we copy a tarball to HDFS, we need to figure out the destination, e.g., /hdp/apps/2.2.1.0-2260/tez
That version number has to come from somewhere, so we use hdp-select status {component_name}, where component name is the 2nd argument.
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
<https://reviews.apache.org/r/30899/#comment117991>
Will clean
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
<https://reviews.apache.org/r/30899/#comment117993>
When hadoop-client is repointed, that means that tez has been upaded as well. This is important during a rolling upgrade, so that we copy the tez tarball to the appropriate folder in HDFS.
- Alejandro Fernandez
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72142
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Feb. 11, 2015, 9:40 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 9:40 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Hitesh Shah <hi...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72097
-----------------------------------------------------------
Ship it!
+1 assuming the comment on the pig client change is correct. ( not tested this change though ). Thanks for the patience in handling all the comments.
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
<https://reviews.apache.org/r/30899/#comment118045>
This change is fine as long as tez client is installed on same host as pig client.
- Hitesh Shah
On Feb. 12, 2015, 2:40 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 12, 2015, 2:40 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Hitesh Shah <hi...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72096
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
<https://reviews.apache.org/r/30899/#comment118041>
Shouldn't this be retained due to the upgrade scenario?
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
<https://reviews.apache.org/r/30899/#comment118042>
Likewise retain upload as hive depends on tez
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
<https://reviews.apache.org/r/30899/#comment118043>
Likewise, when installing pig client ( as pig depends on tez ), tez tarball should be uploaded unless tez client is guaranteed to be installed before pig client.
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
<https://reviews.apache.org/r/30899/#comment118044>
In earlier versions of Tez, tez required its jars to be uploaded to HDFS, does that remain unchanged?
- Hitesh Shah
On Feb. 12, 2015, 2:40 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 12, 2015, 2:40 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 2:40 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
Bugs: AMBARI-9585
https://issues.apache.org/jira/browse/AMBARI-9585
Repository: ambari
Description
-------
Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
Instead, this should happen whenever Tez is installed.
For a RU, the install of new bits should also copy the tarball to HDFS.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
Diff: https://reviews.apache.org/r/30899/diff/
Testing
-------
Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
Then ensured that the tez tarball was copied to hdfs,
su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
Unit tests are in progress.
Thanks,
Alejandro Fernandez
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 2:19 a.m.)
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
Changes
-------
Re-added copying the tez tarball when HiveServer2 is started.
Bugs: AMBARI-9585
https://issues.apache.org/jira/browse/AMBARI-9585
Repository: ambari
Description
-------
Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
Instead, this should happen whenever Tez is installed.
For a RU, the install of new bits should also copy the tarball to HDFS.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
Diff: https://reviews.apache.org/r/30899/diff/
Testing
-------
Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
Then ensured that the tez tarball was copied to hdfs,
su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
Unit tests are in progress.
Thanks,
Alejandro Fernandez
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Feb. 11, 2015, 11:32 p.m., Hitesh Shah wrote:
> > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py, line 38
> > <https://reviews.apache.org/r/30899/diff/1/?file=861155#file861155line38>
> >
> > Why does the location of the tez tarball depend on the version of hadoop client?
The source location is always the same, however, the destination folder in HDFS contains a version number in the name, which is needed for side-by-side versions.
E.g., /hdp/apps/2.2.0.0-2041/tez or /hdp/apps/2.2.0.0-2260/tez
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72051
-----------------------------------------------------------
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 30899: Ambari does not upload Tez tarball to HDFS
when Tez is installed
Posted by Hitesh Shah <hi...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30899/#review72051
-----------------------------------------------------------
I do not have the full context on the code so please bear with any howlers in the comments.
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
<https://reviews.apache.org/r/30899/#comment117982>
What does this line do? What is the relationship between mapreduce and hive-server2? Also, how does the tez.user come into play for mapreduce?
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
<https://reviews.apache.org/r/30899/#comment117984>
trailing whitespace needs cleanup
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
<https://reviews.apache.org/r/30899/#comment117986>
Why does the location of the tez tarball depend on the version of hadoop client?
- Hitesh Shah
On Feb. 11, 2015, 10:42 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30899/
> -----------------------------------------------------------
>
> (Updated Feb. 11, 2015, 10:42 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Tom Beerbower.
>
>
> Bugs: AMBARI-9585
> https://issues.apache.org/jira/browse/AMBARI-9585
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Today, the tez tarball is copied to HDFS only when HiveServer2 is installed, and when the Pig Service check runs.
> Instead, this should happen whenever Tez is installed.
> For a RU, the install of new bits should also copy the tarball to HDFS.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py abfde14
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py 7137e60
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py 00375d7
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 1fc3fbf
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py 3d74113
>
> Diff: https://reviews.apache.org/r/30899/diff/
>
>
> Testing
> -------
>
> Installed a 3-node cluster with HDFS, ZK, MR, YARN, and Tez.
> Then ensured that the tez tarball was copied to hdfs,
>
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.0.0-2041/tez'
>
> When I installed Pig client, on the same host that had the tez client, I was able to run a Pig job, using the following example, http://hortonworks.com/hadoop-tutorial/faster-pig-tez/
>
> I then performed a Rolling Upgrade and the newer tez tarball was indeed copied to HDFS.
> su - hdfs -c 'hdfs dfs -ls /hdp/apps/2.2.1.0-2260/tez'
>
> Unit tests are in progress.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>