You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Dale Bradman <da...@profusion.com> on 2017/04/20 16:45:47 UTC
Express Upgrade to HDP 2.5 fails on restarting the Hive Metastore
Hi.
I'm on Ambari 2.4.2.0-136 and HDP 2.4.2.0 with a view to doing an EXPRESS Upgrade to HDP-2.5.3.0.
It finishes the service checks and the starts restarting Hive where it fails on "Restarting Hive Metastore"
The error is:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 259, in <module>
HiveMetastore().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 696, in restart
self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 114, in pre_upgrade_restart
self.upgrade_schema(env)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 198, in upgrade_schema
status_params.tmp_dir)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py", line 208, in cached_kinit_executor
if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") > timedelta(minutes=expiration_time)):
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d %H:%M:%S.%f'
Doing some digging, the /usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py does not handle non millisecond timestamps (line 205).
The timestamp it is trying to analyse here is the /var/lib/ambari-agent/tmp/kinit_executor_cache/<<hash(hive.service.keytab|hive.principal.name)>>. The timestamp within this file matches that of the error: "'2016-12-19 22:37:33'" - which contains no miliseconds. There are other files in this directory all of which have timesstamps with miliseconds, so I'm going to assume this is 1 in a million. Could you confirm this please?
Options for fixing in my opinion are:
1. Adding an additional conditional statement before line 205 that determines if the timestamp contains miliseconds
2. Manually changing '2016-12-19 22:37:33' to '2016-12-19 22:37:33.000000'
This is an Express Upgrade, so downtime has been arranged but it's reaching critical point, would option 2 work for immediate impact?
I've also upgraded to Ambari 2.5.0.3-7 on a test cluster and the code remains as it is.
Let me know if JIRA required.
Cheers,
Dale
Re: Express Upgrade to HDP 2.5 fails on restarting the Hive
Metastore
Posted by Alejandro Fernandez <af...@hortonworks.com>.
Hi Dale, good find. Feel free to submit a patch for this for branch-2.5 and trunk.
I'm in favor or appending ".0" for ms if last_run_time doesn't already contain ms.
>>> from datetime import datetime
>>> last_run_time = "2016-12-19 22:37:33"
>>> x = datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d %H:%M:%S.%f'
>>> last_run_time_with_ms = last_run_time if "." in last_run_time else last_run_time + ".0"
>>> print last_run_time_with_ms
2016-12-19 22:37:33.0
>>> x = datetime.strptime(last_run_time_with_ms, "%Y-%m-%d %H:%M:%S.%f")
>>> print x
2016-12-19 22:37:33
Thanks,
Alejandro
From: Dale Bradman <da...@profusion.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Thursday, April 20, 2017 at 9:45 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Express Upgrade to HDP 2.5 fails on restarting the Hive Metastore
Hi.
I'm on Ambari 2.4.2.0-136 and HDP 2.4.2.0 with a view to doing an EXPRESS Upgrade to HDP-2.5.3.0.
It finishes the service checks and the starts restarting Hive where it fails on "Restarting Hive Metastore"
The error is:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 259, in <module>
HiveMetastore().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 696, in restart
self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 114, in pre_upgrade_restart
self.upgrade_schema(env)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 198, in upgrade_schema
status_params.tmp_dir)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py", line 208, in cached_kinit_executor
if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") > timedelta(minutes=expiration_time)):
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d %H:%M:%S.%f'
Doing some digging, the /usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py does not handle non millisecond timestamps (line 205).
The timestamp it is trying to analyse here is the /var/lib/ambari-agent/tmp/kinit_executor_cache/<<hash(hive.service.keytab|hive.principal.name)>>. The timestamp within this file matches that of the error: "'2016-12-19 22:37:33'" - which contains no miliseconds. There are other files in this directory all of which have timesstamps with miliseconds, so I'm going to assume this is 1 in a million. Could you confirm this please?
Options for fixing in my opinion are:
1. Adding an additional conditional statement before line 205 that determines if the timestamp contains miliseconds
2. Manually changing '2016-12-19 22:37:33' to '2016-12-19 22:37:33.000000'
This is an Express Upgrade, so downtime has been arranged but it's reaching critical point, would option 2 work for immediate impact?
I've also upgraded to Ambari 2.5.0.3-7 on a test cluster and the code remains as it is.
Let me know if JIRA required.
Cheers,
Dale