You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/11/29 16:55:13 UTC
Review Request 64166: History and Hive server start failed during IOP
migration with AttributeError
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64166/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22547
https://issues.apache.org/jira/browse/AMBARI-22547
Repository: ambari
Description
-------
*STR*
# Deployed cluster with Ambari version: 2.4.2.0_IBM_000000 and HDP version: 4.2.5.0-0000 (LZO is enabled in HDFS configs)
# Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
# Register HDP-2.6.4.0-51 version and call remove iop-slect
# Install the bits and start EU
*Result*
History server start failed with below error:
```
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 81, in pre_upgrade_restart
copy_to_hdfs("tez", params.user_group, params.hdfs_user, skip=params.sysprep_skip_copy_tarballs_hdfs)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 464, in copy_to_hdfs
source_file = prepare_function()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 94, in _prepare_tez_tarball
hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
File "/usr/lib64/python2.6/posixpath.py", line 65, in join
if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'
```
The error looks to be due to lzo changes, as the following code has the issue:
```
# if enabled, LZO GPL libraries must be copied as well
91 if lzo_utils.should_install_lzo():
92 stack_root = Script.get_stack_root()
93 tez_version = component_version.get_component_repository_version("TEZ")
94 hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
95
```
**In this system, Tez is not installed and Hive is on MapReduce. However, the Tez binaries are installed and should be updated in the event that Tez is installed after upgrade.**
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py bd1cede834
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 2871dc0ff3
Diff: https://reviews.apache.org/r/64166/diff/1/
Testing
-------
Manual install & upgrade
Thanks,
Jonathan Hurley
Re: Review Request 64166: History and Hive server start failed during
IOP migration with AttributeError
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64166/#review192165
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Nov. 29, 2017, 11:55 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64166/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2017, 11:55 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22547
> https://issues.apache.org/jira/browse/AMBARI-22547
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.4.2.0_IBM_000000 and HDP version: 4.2.5.0-0000 (LZO is enabled in HDFS configs)
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 version and call remove iop-slect
> # Install the bits and start EU
>
> *Result*
> History server start failed with below error:
> ```
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 81, in pre_upgrade_restart
> copy_to_hdfs("tez", params.user_group, params.hdfs_user, skip=params.sysprep_skip_copy_tarballs_hdfs)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 464, in copy_to_hdfs
> source_file = prepare_function()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 94, in _prepare_tez_tarball
> hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
> File "/usr/lib64/python2.6/posixpath.py", line 65, in join
> if b.startswith('/'):
> AttributeError: 'NoneType' object has no attribute 'startswith'
> ```
>
> The error looks to be due to lzo changes, as the following code has the issue:
> ```
> # if enabled, LZO GPL libraries must be copied as well
> 91 if lzo_utils.should_install_lzo():
> 92 stack_root = Script.get_stack_root()
> 93 tez_version = component_version.get_component_repository_version("TEZ")
> 94 hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
> 95
> ```
>
> **In this system, Tez is not installed and Hive is on MapReduce. However, the Tez binaries are installed and should be updated in the event that Tez is installed after upgrade.**
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py bd1cede834
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 2871dc0ff3
>
>
> Diff: https://reviews.apache.org/r/64166/diff/1/
>
>
> Testing
> -------
>
> Manual install & upgrade
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64166: History and Hive server start failed during
IOP migration with AttributeError
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Nov. 29, 2017, 1:41 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> > Lines 96 (patched)
> > <https://reviews.apache.org/r/64166/diff/1/?file=1903726#file1903726line96>
> >
> > seems that tez_version variable is better to rename to something like component_version or just version
That's a good point. I'll rename it.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64166/#review192153
-----------------------------------------------------------
On Nov. 29, 2017, 11:55 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64166/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2017, 11:55 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22547
> https://issues.apache.org/jira/browse/AMBARI-22547
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.4.2.0_IBM_000000 and HDP version: 4.2.5.0-0000 (LZO is enabled in HDFS configs)
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 version and call remove iop-slect
> # Install the bits and start EU
>
> *Result*
> History server start failed with below error:
> ```
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 81, in pre_upgrade_restart
> copy_to_hdfs("tez", params.user_group, params.hdfs_user, skip=params.sysprep_skip_copy_tarballs_hdfs)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 464, in copy_to_hdfs
> source_file = prepare_function()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 94, in _prepare_tez_tarball
> hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
> File "/usr/lib64/python2.6/posixpath.py", line 65, in join
> if b.startswith('/'):
> AttributeError: 'NoneType' object has no attribute 'startswith'
> ```
>
> The error looks to be due to lzo changes, as the following code has the issue:
> ```
> # if enabled, LZO GPL libraries must be copied as well
> 91 if lzo_utils.should_install_lzo():
> 92 stack_root = Script.get_stack_root()
> 93 tez_version = component_version.get_component_repository_version("TEZ")
> 94 hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
> 95
> ```
>
> **In this system, Tez is not installed and Hive is on MapReduce. However, the Tez binaries are installed and should be updated in the event that Tez is installed after upgrade.**
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py bd1cede834
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 2871dc0ff3
>
>
> Diff: https://reviews.apache.org/r/64166/diff/1/
>
>
> Testing
> -------
>
> Manual install & upgrade
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64166: History and Hive server start failed during
IOP migration with AttributeError
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64166/#review192153
-----------------------------------------------------------
Ship it!
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 96 (patched)
<https://reviews.apache.org/r/64166/#comment270177>
seems that tez_version variable is better to rename to something like component_version or just version
- Dmytro Grinenko
On Nov. 29, 2017, 4:55 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64166/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2017, 4:55 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22547
> https://issues.apache.org/jira/browse/AMBARI-22547
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.4.2.0_IBM_000000 and HDP version: 4.2.5.0-0000 (LZO is enabled in HDFS configs)
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 version and call remove iop-slect
> # Install the bits and start EU
>
> *Result*
> History server start failed with below error:
> ```
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 81, in pre_upgrade_restart
> copy_to_hdfs("tez", params.user_group, params.hdfs_user, skip=params.sysprep_skip_copy_tarballs_hdfs)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 464, in copy_to_hdfs
> source_file = prepare_function()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 94, in _prepare_tez_tarball
> hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
> File "/usr/lib64/python2.6/posixpath.py", line 65, in join
> if b.startswith('/'):
> AttributeError: 'NoneType' object has no attribute 'startswith'
> ```
>
> The error looks to be due to lzo changes, as the following code has the issue:
> ```
> # if enabled, LZO GPL libraries must be copied as well
> 91 if lzo_utils.should_install_lzo():
> 92 stack_root = Script.get_stack_root()
> 93 tez_version = component_version.get_component_repository_version("TEZ")
> 94 hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, "hadoop", "lib", "native")
> 95
> ```
>
> **In this system, Tez is not installed and Hive is on MapReduce. However, the Tez binaries are installed and should be updated in the event that Tez is installed after upgrade.**
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py bd1cede834
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 2871dc0ff3
>
>
> Diff: https://reviews.apache.org/r/64166/diff/1/
>
>
> Testing
> -------
>
> Manual install & upgrade
>
>
> Thanks,
>
> Jonathan Hurley
>
>