You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Alejandro Fernandez (JIRA)" <ji...@apache.org> on 2015/02/11 01:14:11 UTC
[jira] [Commented] (AMBARI-9559) RU: Finalize fails because
Secondary Namenode was not deleted during HA Wizard
[ https://issues.apache.org/jira/browse/AMBARI-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315240#comment-14315240 ]
Alejandro Fernandez commented on AMBARI-9559:
---------------------------------------------
I was not able to reproduce this, even after installing the exact same services on the same hosts, and using the same ambari-server version. It could be the case that some thread is holding a reference to the object that is being deleted, so it persists. So I added another check to PreUpgradeCheckResourceProvider that ensures that SECONDARY_NAMENODE is not present on any hosts.
> RU: Finalize fails because Secondary Namenode was not deleted during HA Wizard
> ------------------------------------------------------------------------------
>
> Key: AMBARI-9559
> URL: https://issues.apache.org/jira/browse/AMBARI-9559
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.0.0
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Fix For: 2.0.0
>
>
> While performing a RU, it failed in Finalize because the SECONDARY_NAMENODE component was still on an old version, since it was not deleted by the Namenode HA wizard.
> This was a 3-node cluster with HDFS, MR, YARN, TEZ, Hive, ZK, Pig, AMS using HDP 2.2.0.0 GA.
> {code}
> ambari=> select component_name, version, current_state, upgrade_state, host_name from hostcomponentstate ORDER BY host_name, component_name;
> component_name | version | current_state | upgrade_state | host_name
> ---------------------+--------------+---------------+---------------+-------------------------
> DATANODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6401.ambari.apache.org
> HCAT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> HDFS_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> HIVE_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> JOURNALNODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6401.ambari.apache.org
> MAPREDUCE2_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> METRIC_MONITOR | UNKNOWN | STARTED | NONE | c6401.ambari.apache.org
> NAMENODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6401.ambari.apache.org
> NODEMANAGER | 2.2.1.0-2270 | STARTED | COMPLETE | c6401.ambari.apache.org
> PIG | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> TEZ_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> YARN_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> ZKFC | UNKNOWN | STARTED | NONE | c6401.ambari.apache.org
> ZOOKEEPER_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6401.ambari.apache.org
> ZOOKEEPER_SERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6401.ambari.apache.org
> APP_TIMELINE_SERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> DATANODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> HCAT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> HDFS_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> HISTORYSERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> HIVE_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> HIVE_METASTORE | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> HIVE_SERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> JOURNALNODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> MAPREDUCE2_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> METRIC_MONITOR | UNKNOWN | STARTED | NONE | c6402.ambari.apache.org
> MYSQL_SERVER | UNKNOWN | STARTED | NONE | c6402.ambari.apache.org
> NAMENODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> NODEMANAGER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> PIG | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> RESOURCEMANAGER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> SECONDARY_NAMENODE | 2.2.0.0-2041 | INSTALLED | NONE | c6402.ambari.apache.org
> TEZ_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> WEBHCAT_SERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> YARN_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> ZKFC | UNKNOWN | STARTED | NONE | c6402.ambari.apache.org
> ZOOKEEPER_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6402.ambari.apache.org
> ZOOKEEPER_SERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6402.ambari.apache.org
> DATANODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6403.ambari.apache.org
> HCAT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> HDFS_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> HIVE_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> JOURNALNODE | 2.2.1.0-2270 | STARTED | COMPLETE | c6403.ambari.apache.org
> MAPREDUCE2_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> METRIC_COLLECTOR | UNKNOWN | STARTED | NONE | c6403.ambari.apache.org
> METRIC_MONITOR | UNKNOWN | STARTED | NONE | c6403.ambari.apache.org
> NODEMANAGER | 2.2.1.0-2270 | STARTED | COMPLETE | c6403.ambari.apache.org
> PIG | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> TEZ_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> YARN_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> ZOOKEEPER_CLIENT | 2.2.1.0-2270 | INSTALLED | COMPLETE | c6403.ambari.apache.org
> ZOOKEEPER_SERVER | 2.2.1.0-2270 | STARTED | COMPLETE | c6403.ambari.apache.org
> (52 rows)
> {code}
> {code}
> ambari=> select hv.host_name, hv.state, rv.version from host_version AS hv JOIN repo_version AS rv ON hv.repo_version_id = rv.repo_version_id ORDER BY rv.repo_version_id, hv.host_name;
> host_name | state | version
> -------------------------+-----------+--------------
> c6401.ambari.apache.org | CURRENT | 2.2.0.0-2041
> c6402.ambari.apache.org | CURRENT | 2.2.0.0-2041
> c6403.ambari.apache.org | CURRENT | 2.2.0.0-2041
> c6401.ambari.apache.org | UPGRADED | 2.2.1.0-2270
> c6402.ambari.apache.org | UPGRADING | 2.2.1.0-2270
> c6403.ambari.apache.org | UPGRADED | 2.2.1.0-2270
> (6 rows)
> {code}
> ambari-server logs do not show anything suspicious,
> {code}
> 20:54:24,755 INFO [qtp960053895-1932] HeartBeatHandler:505 - Updating applied config on service HDFS, component ZKFC, host c6402.ambari.apache.org
> 20:54:25,579 INFO [qtp960053895-1932] HeartBeatHandler:505 - Updating applied config on service HDFS, component ZKFC, host c6401.ambari.apache.org
> 20:54:26,844 INFO [qtp1581173354-1633] ServiceComponentImpl:592 - Deleting servicecomponenthost for cluster, clusterName=MyCluster, serviceName=HDFS, componentName=SECONDARY_NAMENODE, hostname=c6402.ambari.apache.org
> 20:54:26,909 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=HDFS, request=clusterName=MyCluster, serviceName=HDFS, desiredState=STARTED
> 20:54:26,909 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=MAPREDUCE2, request=clusterName=MyCluster, serviceName=MAPREDUCE2, desiredState=STARTED
> 20:54:26,909 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=TEZ, request=clusterName=MyCluster, serviceName=TEZ, desiredState=STARTED
> 20:54:26,910 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=AMS, request=clusterName=MyCluster, serviceName=AMS, desiredState=STARTED
> 20:54:26,913 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=ZOOKEEPER, request=clusterName=MyCluster, serviceName=ZOOKEEPER, desiredState=STARTED
> 20:54:26,913 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=YARN, request=clusterName=MyCluster, serviceName=YARN, desiredState=STARTED
> 20:54:26,913 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=PIG, request=clusterName=MyCluster, serviceName=PIG, desiredState=STARTED
> 20:54:26,914 INFO [qtp1581173354-1914] AbstractResourceProvider:616 - Received a updateService request, clusterName=MyCluster, serviceName=HIVE, request=clusterName=MyCluster, serviceName=HIVE, desiredState=STARTED
> 20:54:26,917 INFO [qtp1581173354-1914] AmbariManagementControllerImpl:1581 - Client hosts for reinstall : 8
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)