You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Irina Easterling (JIRA)" <ji...@apache.org> on 2015/12/21 23:44:46 UTC
[jira] [Commented] (AMBARI-14455) After changing the hostname on
the HA cluster the ZKFC's metadata still uses old hostname.
[ https://issues.apache.org/jira/browse/AMBARI-14455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067194#comment-15067194 ]
Irina Easterling commented on AMBARI-14455:
-------------------------------------------
System version:
{code}
Versions
+-----------------------+-----------+
| Name | Version |
+-----------------------+-----------+
| HDP | 2.3 |
| AMBARI_SERVER | 2.2.0.0 |
ambari-metrics-monitor-2.2.0.0-1308
ambari-server-2.2.0.0-1308
ambari-agent-2.2.0.0-1308
ambari-metrics-collector-2.2.0.0-1308
ambari-metrics-hadoop-sink-2.2.0.0-1308
{code}
Cluster configuration: 3Master, 3 DataNodes, 1Edge
Steps to reproduce:
Change hostname on HA cluster following instruction in (https://issues.apache.org/jira/browse/AMBARI-13507)
1. Stop all pending commands/jobs
2. Stop all services via Ambari
3. Backup Ambari db.
4. ssh master1 and stop ambari-server and all ambari-agents
5. Create json file with the host names changes.
{code}
{
"VM4C8" : {
"selena1.labs.teradata.com" : "hostnamechange1.labs.teradata.com",
"selena2.labs.teradata.com" : "hostnamechange2.labs.teradata.com",
"selena3.labs.teradata.com" : "hostnamechange3.labs.teradata.com",
"selena4.labs.teradata.com" : "hostnamechange4.labs.teradata.com",
"selena5.labs.teradata.com" : "hostnamechange5.labs.teradata.com",
"selena6.labs.teradata.com" : "hostnamechange6.labs.teradata.com",
"selena8.labs.teradata.com" : "hostnamechange8.labs.teradata.com"
}
}
{code}
6.Execute command on the ambari-server hosts (master1):
{code}
ambari-server update-host-names host_names_changes.json
{code}
7. Change etc/hosts, hostname, /etc/HOSTNAME on all nodes according to changes that added to json file
8. Change 'ambari-agent.ini' file hostname field with new hostname
{code}
hostname=hostnamechange1.labs.teradata.com
{code}
9. Start ambar-server and all ambari-agents
10. Start all serivces via Ambari
Results:Services failed to start
The issue is that the old hostname is in ZKFC's metadata. This is from/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-zkfc-hostnamechange1.log:
{code}
2015-12-21 13:41:07,941 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:becomeActive(863)) - Exception handling the winning of election
java.lang.RuntimeException: Mismatched address stored in ZK for NameNode at hostnamechange1.labs.teradata.com/39.0.64.1:8020: Stored protobuf was nameser
viceId: "VM4C8"
namenodeId: "nn1"
hostname: "selena1.labs.teradata.com"
port: 8020
zkfcPort: 8019
, address from our own configuration for this NameNode was hostnamechange1.labs.teradata.com/39.0.64.1:8020
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.dataToTarget(DFSZKFailoverController.java:75)
at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:502)
at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61)
at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892)
at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:956)
at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:855)
at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
{code}
> After changing the hostname on the HA cluster the ZKFC's metadata still uses old hostname.
> ------------------------------------------------------------------------------------------
>
> Key: AMBARI-14455
> URL: https://issues.apache.org/jira/browse/AMBARI-14455
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Reporter: Irina Easterling
>
> After changing the hostname to be new on the HA cluster the ZKFC's metadata still uses old hostname.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)