You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2014/10/18 03:17:35 UTC
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------
(Updated Oct. 18, 2014, 1:17 a.m.)
Review request for Ambari, Sumit Mohanty and Sid Wagle.
Bugs: AMBARI-7825
https://issues.apache.org/jira/browse/AMBARI-7825
Repository: ambari
Description
-------
HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files
This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py b034cbc
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
Diff: https://reviews.apache.org/r/26858/diff/
Testing
-------
Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
3. Copy changed files
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
6. Before running Tez script, check that it still has the dynamic property without the actual values,
less /etc/tez/conf/tez-site.xml
/ tez.lib.uris
7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;
*Unit tests currently fail*
Thanks,
Alejandro Fernandez
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/#review57388
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
<https://reviews.apache.org/r/26858/#comment98075>
We were using the 2.0.6 and 2.1 scripts to hold 2.2 stack-logic, which is incorrect; so I moved this to 2.2-specific scripts.
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
<https://reviews.apache.org/r/26858/#comment98076>
None of these should be in 2.0.6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive.py
<https://reviews.apache.org/r/26858/#comment98078>
This file was copied from 2.0.6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_server.py
<https://reviews.apache.org/r/26858/#comment98079>
This file was copied from 2.0.6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_service.py
<https://reviews.apache.org/r/26858/#comment98080>
This file was copied from 2.0.6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/params.py
<https://reviews.apache.org/r/26858/#comment98081>
This file was copied from 2.0.6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/status_params.py
<https://reviews.apache.org/r/26858/#comment98082>
This file was copied from 2.0.6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/webhcat.py
<https://reviews.apache.org/r/26858/#comment98085>
This TODO comment is new.
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/params.py
<https://reviews.apache.org/r/26858/#comment98088>
Half of this file was copied from 2.1.
Lines 27 through 38 are new, and also 49-60
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/params.py
<https://reviews.apache.org/r/26858/#comment98087>
Half of this file was copied from 2.1.
The HdfsDirectory partial function is new.
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez.py
<https://reviews.apache.org/r/26858/#comment98089>
This file was copied from 2.1,
lines 36 - 44 are new (plus the imports).
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez_client.py
<https://reviews.apache.org/r/26858/#comment98090>
This file was copied from 2.1
- Alejandro Fernandez
On Oct. 20, 2014, 6:36 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26858/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 6:36 p.m.)
>
>
> Review request for Ambari, Sumit Mohanty and Sid Wagle.
>
>
> Bugs: AMBARI-7825
> https://issues.apache.org/jira/browse/AMBARI-7825
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HDP 2.2 using versioned RPMs
> When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
> (Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
> The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
> Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
>
> Instead, it should preserve the version number when copying to HDFS.
> Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
>
> It does 2 things:
> 1. Copies files from the local file system to HDFS using a predefined directory structure
> 2. Provides a way of doing dynamic variable interpretation in *-site.xml files
>
> This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_service.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/status_params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/webhcat.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/params.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez_client.py PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
> ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
> ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
>
> Diff: https://reviews.apache.org/r/26858/diff/
>
>
> Testing
> -------
>
> Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
>
>
> 1. Set properties
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
>
> 2. Verified properties were saved
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
>
> 3. Copy changed files
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
> rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
>
> 4. Check that tarballs are not already in HDFS. If they are, delete them.
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
>
> 5. Restart Hive service and verify file is copied to HDFS,
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
>
> 6. Before running Tez script, check that it still has the dynamic property without the actual values,
> less /etc/tez/conf/tez-site.xml
> / tez.lib.uris
>
> 7. Run tez script manually, which generates tez-site.xml during its install
> Need to first find the latest command file.
> ls -ltr /var/lib/ambari-agent/data/command-*.json
> Get the digits from the last command-*.json file
> python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
>
> Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
> <name>tez.lib.uris</name>
> <value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
>
> 8. Ensure Hive runs with tez-execution engine
> [root@c6404 ~]# su - hive
> [hive@c6404 ~]$ hive
> hive> select name FROM school order by name asc limit 10;
>
> *Unit tests passed*
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/#review57460
-----------------------------------------------------------
Ship it!
Ship It!
- Sid Wagle
On Oct. 20, 2014, 10:26 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26858/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 10:26 p.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Mahadev Konar, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Sid Wagle.
>
>
> Bugs: AMBARI-7825
> https://issues.apache.org/jira/browse/AMBARI-7825
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HDP 2.2 using versioned RPMs
> When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
> (Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
> The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
> Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
>
> Instead, it should preserve the version number when copying to HDFS.
> Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
>
> It does 2 things:
> 1. Copies files from the local file system to HDFS using a predefined directory structure
> 2. Provides a way of doing dynamic variable interpretation in *-site.xml files
>
> This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py b034cbc
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
> ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
> ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
>
> Diff: https://reviews.apache.org/r/26858/diff/
>
>
> Testing
> -------
>
> Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
>
>
> 1. Set properties
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
>
> 2. Verified properties were saved
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
>
> 3. Copy changed files
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
> rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
>
> 4. Check that tarballs are not already in HDFS. If they are, delete them.
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
>
> 5. Restart Hive service and verify file is copied to HDFS,
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
>
> 6. Before running Tez script, check that it still has the dynamic property without the actual values,
> less /etc/tez/conf/tez-site.xml
> / tez.lib.uris
>
> 7. Run tez script manually, which generates tez-site.xml during its install
> Need to first find the latest command file.
> ls -ltr /var/lib/ambari-agent/data/command-*.json
> Get the digits from the last command-*.json file
> python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
>
> Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
> <name>tez.lib.uris</name>
> <value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
>
> 8. Ensure Hive runs with tez-execution engine
> [root@c6404 ~]# su - hive
> [hive@c6404 ~]$ hive
> hive> select name FROM school order by name asc limit 10;
>
> *Unit tests passed*
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/#review57458
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Oct. 20, 2014, 6:26 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26858/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 6:26 p.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Mahadev Konar, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Sid Wagle.
>
>
> Bugs: AMBARI-7825
> https://issues.apache.org/jira/browse/AMBARI-7825
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HDP 2.2 using versioned RPMs
> When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
> (Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
> The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
> Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
>
> Instead, it should preserve the version number when copying to HDFS.
> Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
>
> It does 2 things:
> 1. Copies files from the local file system to HDFS using a predefined directory structure
> 2. Provides a way of doing dynamic variable interpretation in *-site.xml files
>
> This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py b034cbc
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
> ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
> ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
>
> Diff: https://reviews.apache.org/r/26858/diff/
>
>
> Testing
> -------
>
> Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
>
>
> 1. Set properties
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
>
> 2. Verified properties were saved
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
>
> 3. Copy changed files
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
> rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
>
> 4. Check that tarballs are not already in HDFS. If they are, delete them.
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
>
> 5. Restart Hive service and verify file is copied to HDFS,
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
>
> 6. Before running Tez script, check that it still has the dynamic property without the actual values,
> less /etc/tez/conf/tez-site.xml
> / tez.lib.uris
>
> 7. Run tez script manually, which generates tez-site.xml during its install
> Need to first find the latest command file.
> ls -ltr /var/lib/ambari-agent/data/command-*.json
> Get the digits from the last command-*.json file
> python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
>
> Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
> <name>tez.lib.uris</name>
> <value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
>
> 8. Ensure Hive runs with tez-execution engine
> [root@c6404 ~]# su - hive
> [hive@c6404 ~]$ hive
> hive> select name FROM school order by name asc limit 10;
>
> *Unit tests passed*
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/#review57457
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Oct. 20, 2014, 10:26 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26858/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 10:26 p.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Mahadev Konar, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Sid Wagle.
>
>
> Bugs: AMBARI-7825
> https://issues.apache.org/jira/browse/AMBARI-7825
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HDP 2.2 using versioned RPMs
> When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
> (Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
> The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
> Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
>
> Instead, it should preserve the version number when copying to HDFS.
> Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
>
> It does 2 things:
> 1. Copies files from the local file system to HDFS using a predefined directory structure
> 2. Provides a way of doing dynamic variable interpretation in *-site.xml files
>
> This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py b034cbc
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
> ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
> ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
>
> Diff: https://reviews.apache.org/r/26858/diff/
>
>
> Testing
> -------
>
> Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
>
>
> 1. Set properties
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
>
> 2. Verified properties were saved
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
>
> 3. Copy changed files
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
> rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
>
> 4. Check that tarballs are not already in HDFS. If they are, delete them.
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
>
> 5. Restart Hive service and verify file is copied to HDFS,
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
>
> 6. Before running Tez script, check that it still has the dynamic property without the actual values,
> less /etc/tez/conf/tez-site.xml
> / tez.lib.uris
>
> 7. Run tez script manually, which generates tez-site.xml during its install
> Need to first find the latest command file.
> ls -ltr /var/lib/ambari-agent/data/command-*.json
> Get the digits from the last command-*.json file
> python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
>
> Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
> <name>tez.lib.uris</name>
> <value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
>
> 8. Ensure Hive runs with tez-execution engine
> [root@c6404 ~]# su - hive
> [hive@c6404 ~]$ hive
> hive> select name FROM school order by name asc limit 10;
>
> *Unit tests passed*
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------
(Updated Oct. 20, 2014, 10:26 p.m.)
Review request for Ambari, Dmytro Sen, Jonathan Hurley, Mahadev Konar, Nate Cole, Sumit Mohanty, Srimanth Gunturi, and Sid Wagle.
Bugs: AMBARI-7825
https://issues.apache.org/jira/browse/AMBARI-7825
Repository: ambari
Description
-------
HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files
This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py b034cbc
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
Diff: https://reviews.apache.org/r/26858/diff/
Testing
-------
Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
3. Copy changed files
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
6. Before running Tez script, check that it still has the dynamic property without the actual values,
less /etc/tez/conf/tez-site.xml
/ tez.lib.uris
7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;
*Unit tests passed*
Thanks,
Alejandro Fernandez
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------
(Updated Oct. 20, 2014, 10:02 p.m.)
Review request for Ambari, Sumit Mohanty and Sid Wagle.
Changes
-------
Reverted to keeping scripts inside 2.0.6 and 2.1 stack, but still work with HDP 2.2
Bugs: AMBARI-7825
https://issues.apache.org/jira/browse/AMBARI-7825
Repository: ambari
Description
-------
HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files
This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py b034cbc
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
Diff: https://reviews.apache.org/r/26858/diff/
Testing
-------
Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
3. Copy changed files
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
6. Before running Tez script, check that it still has the dynamic property without the actual values,
less /etc/tez/conf/tez-site.xml
/ tez.lib.uris
7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;
*Unit tests passed*
Thanks,
Alejandro Fernandez
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------
(Updated Oct. 20, 2014, 6:36 p.m.)
Review request for Ambari, Sumit Mohanty and Sid Wagle.
Bugs: AMBARI-7825
https://issues.apache.org/jira/browse/AMBARI-7825
Repository: ambari
Description
-------
HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files
This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_server.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_service.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/status_params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/webhcat.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez_client.py PRE-CREATION
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
Diff: https://reviews.apache.org/r/26858/diff/
Testing (updated)
-------
Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
3. Copy changed files
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
6. Before running Tez script, check that it still has the dynamic property without the actual values,
less /etc/tez/conf/tez-site.xml
/ tez.lib.uris
7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;
*Unit tests passed*
Thanks,
Alejandro Fernandez
Re: Review Request 26858: Rolling Upgrades -
hdfs:///apps/tez/tez.tar.gz needs to be versioned
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------
(Updated Oct. 20, 2014, 6:35 p.m.)
Review request for Ambari, Sumit Mohanty and Sid Wagle.
Changes
-------
Copied scripts to 2.2 folder.
Bugs: AMBARI-7825
https://issues.apache.org/jira/browse/AMBARI-7825
Repository: ambari
Description
-------
HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed, and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still needs to reference the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files
This will break the Download Client Configs, which calculates the dynamic properties on the agent-side.
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py 453c546
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 34f2d96
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py 08a0a50
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py 5834b89
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_server.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/hive_service.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/status_params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/package/scripts/webhcat.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 6a793bf
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/params.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/package/scripts/tez_client.py PRE-CREATION
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac
ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346
ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9
Diff: https://reviews.apache.org/r/26858/diff/
Testing
-------
Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
3. Copy changed files
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
6. Before running Tez script, check that it still has the dynamic property without the actual values,
less /etc/tez/conf/tez-site.xml
/ tez.lib.uris
7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;
*Unit tests currently fail*
Thanks,
Alejandro Fernandez