You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "suja s (JIRA)" <ji...@apache.org> on 2012/05/30 16:16:23 UTC
[jira] [Created] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
suja s created HDFS-3477:
----------------------------
Summary: FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
Key: HDFS-3477
URL: https://issues.apache.org/jira/browse/HDFS-3477
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: suja s
Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the conenction is completely established. This leads to failure at the subsequent point if conenction was not complete by then.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Status: Open (was: Patch Available)
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Attachment: HDFS-3477.patch
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287486#comment-13287486 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530557/HDFS-3477.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2565//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2565//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293374#comment-13293374 ]
Rakesh R commented on HDFS-3477:
--------------------------------
Thanks Todd. I have updated the patch by fixing the comment.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511170#comment-13511170 ]
Aaron T. Myers commented on HDFS-3477:
--------------------------------------
+1, the latest patch looks good to me.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.0-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch, hdfs-3477.txt, hdfs-3477.txt
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288499#comment-13288499 ]
Rakesh R commented on HDFS-3477:
--------------------------------
Thanks Todd for the comments. I'll rework and upload the patch.
For the following comment:
{quote}In the case that the ZK connection fails (eg ZK is just not listening) do we now have to wait for the full session timeout? I think the latch should fire on either SyncConnected or Disconnected events, not just on SyncConnected.
We probably want some kind of sanity check that we don't get any other kind of events in the ZKConnectionWatcher. Otherwise I'm afraid we might end up swallowing an event inadvertently, which could result in the elector hanging. In that case, we should also make sure that the "authenticated" message we get from a SASL-capable ZK is in the list of acceptable events.
{quote}
I'm just thinking to change ZKConnectionWatcher logic like, it will just wait for any events from ZK and pass it to the activestandbyelector.processWatchEvent(zk, event). Here it will be processed those events. With this we would be able to avoid sanity checks in different places.
Activestandbyelector.processWatchEvent(zk, event) is considering only the following events and there is no special sasl checks exists.
case SyncConnected:
case Disconnected:
case Expired:
default:
Could you please tell me your thoughts on this.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287361#comment-13287361 ]
Rakesh R commented on HDFS-3477:
--------------------------------
Attached patch. I have reused zkSessionTimeout as waiting period for zk connection establishment.
Please review.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "suja s (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
suja s updated HDFS-3477:
-------------------------
Description: Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then. (was: Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the conenction is completely established. This leads to failure at the subsequent point if conenction was not complete by then.)
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285699#comment-13285699 ]
Rakesh R commented on HDFS-3477:
--------------------------------
ActiveStandbyElector is missing the checks for SyncConnected event and not waiting for the connection establishment. I think we would be having a connection timeout associated with zk client and if it doesn't get the connection after this period throw exception saying 'unable to establish connection with zk server'
{noformat}
protected synchronized ZooKeeper getNewZooKeeper() throws IOException {
ZooKeeper zk = new ZooKeeper(zkHostPort, zkSessionTimeout, null);
zk.register(new WatcherWithClientRef(zk));
.......
.......
}
{noformat}
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501358#comment-13501358 ]
Rakesh R commented on HDFS-3477:
--------------------------------
@Todd
Sorry for the late reply. Yeah, you are absolutely correct. I had missed the case of receiving multiple events in between wait completion and watcher registration.
I also could see a very corner case like:
Recevived the ZK connection event, immediately before the watcher registration, say ZK got Disconnected the event will be received to the old watcher and will never execute this event. In this case would be still in Synconnected and could go to an inconsistent state. I have tried this scenario in my cluster, but not able to re-produce the same, finally I just tried keeping debug points and has re-produced. Your approach of queuingup the events is fine and this is executing all the events sequentially without missing anything.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.0-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch, hdfs-3477.txt, hdfs-3477.txt
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Attachment: HDFS-3477.3.patch
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286930#comment-13286930 ]
Todd Lipcon commented on HDFS-3477:
-----------------------------------
Do you want to take a crack at a patch, Rakesh?
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Status: Patch Available (was: Open)
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3477:
------------------------------
Attachment: hdfs-3477.txt
Fix inconsistent synchronization findbugs - just had to add a synchronized block around an access to the {{zk}} variable in the watcher
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.0-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch, hdfs-3477.txt, hdfs-3477.txt
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Status: Open (was: Patch Available)
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Status: Patch Available (was: Open)
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Status: Patch Available (was: Open)
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R reassigned HDFS-3477:
------------------------------
Assignee: Rakesh R
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: suja s
> Assignee: Rakesh R
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the conenction is completely established. This leads to failure at the subsequent point if conenction was not complete by then.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3477:
------------------------------
Attachment: hdfs-3477.txt
Hey Rakesh,
I modified your patch a bit and came up with this new rev. I'm afraid there might have been a race in the old version -- if multiple events came in to the registration watcher, it might have missed forwarding the second to the main watcher, which could produce an inconsistent state. I changed it around a bit so that the existing watcher wrapper takes care of the latching.
If you're able to reproduce this in a real cluster, can you give it a try? I've never seen the problem in practice, but my patch passes the unit test you added.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.0-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch, hdfs-3477.txt
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Attachment: HDFS-3477.2.patch
Attached latest patch by fixing the comments. Please review.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291942#comment-13291942 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531384/HDFS-3477.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2621//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2621//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3477:
------------------------------
Affects Version/s: (was: 3.0.0)
2.0.1-alpha
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293590#comment-13293590 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531773/HDFS-3477.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2643//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2643//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287734#comment-13287734 ]
Todd Lipcon commented on HDFS-3477:
-----------------------------------
Nice work. Thanks for fixing this. A few comments below:
- When you catch and swallow InterruptedException, you should call {{Thread.currentThread().interrupt()}} afterward, so that the interrupt status isn't lost
----
{code}
+ /**
+ * Watcher for waiting(connectionTimeout in milliseconds) to establish
+ * connection with the zookeeper server.
+ */
{code}
I don't think you need the parenthetical ("connectionTimeout in milliseconds") since you already specify that down below in the relevant javadoc.
----
- In the case that the ZK connection fails (eg ZK is just not listening) do we now have to wait for the full session timeout? I think the latch should fire on either SyncConnected or Disconnected events, not just on SyncConnected.
- We probably want some kind of sanity check that we don't get any other kind of events in the ZKConnectionWatcher. Otherwise I'm afraid we might end up swallowing an event inadvertently, which could result in the elector hanging. In that case, we should also make sure that the "authenticated" message we get from a SASL-capable ZK is in the list of acceptable events.
----
- Nit: can you make ZKConnectionWatcher private, and move it to the bottom of the file? Also, I think zkConnectLatch can be made private.
- Test nit: can you please use the {{GenericTestUtils.assertExceptionContains()}} utility to check the exception? I've found that the JUnit "expected exception" trick isn't great, since it catches all sorts of exceptions instead of just the targeted error you want to verify
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495484#comment-13495484 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553145/hdfs-3477.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3484//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3484//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.0-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch, hdfs-3477.txt, hdfs-3477.txt
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Attachment: HDFS-3477.3.patch
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412675#comment-13412675 ]
Rakesh R commented on HDFS-3477:
--------------------------------
Added links to HDFS-3635 as I feel the cause is same and failing after timeout:
{code}java.lang.Exception: test timed out after 30000 milliseconds
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.ha.ZKFailoverController.waitForActiveAttempt(ZKFailoverController.java:457)
at org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:645)
at org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:58)
at org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:590)
at org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:587){code}
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "suja s (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
suja s updated HDFS-3477:
-------------------------
Description:
Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
Exception trace for format
{noformat}
12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
... 8 more
{noformat}
was:Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rakesh R updated HDFS-3477:
---------------------------
Attachment: HDFS-3477.1.patch
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494557#comment-13494557 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12552945/hdfs-3477.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3479//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3479//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3479//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.0-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch, hdfs-3477.txt
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402121#comment-13402121 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533626/HDFS-3477.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2711//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2711//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.3.patch, HDFS-3477.3.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292109#comment-13292109 ]
Todd Lipcon commented on HDFS-3477:
-----------------------------------
Hey Rakesh. Looking good. One more small comment which I missed:
- We currently have the following code lower down inside {{doRun()}}:
{code}
if (ioe.getCause() instanceof KeeperException.ConnectionLossException) {
LOG.fatal("Unable to start failover controller. Unable to connect " +
"to ZooKeeper quorum at " + zkQuorum + ". Please check the " +
"configured value for " + ZK_QUORUM_KEY + " and ensure that " +
"ZooKeeper is running.");
return ERR_CODE_NO_ZK;
} else {
throw ioe;
}
{code}
but with your bug fix, we'll never get here since {{initZK()}} would have failed already. So, can you move this fatal error message up to where you catch {{KeeperException}} around {{initZK()}}? This way we'll give a nice message for users who have a mistake in their configuration. Thanks.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 2.0.1-alpha
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.1.patch, HDFS-3477.2.patch, HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3477) FormatZK and ZKFC startup can fail
due to zkclient connection establishment delay
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287383#comment-13287383 ]
Hadoop QA commented on HDFS-3477:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530543/HDFS-3477.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2564//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2564//console
This message is automatically generated.
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
> Attachments: HDFS-3477.patch
>
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
> Exception trace for format
> {noformat}
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Socket connection established to HOST-xx-xx-xx-55/xx.xx.xx.55:2182, initiating session
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: Session establishment complete on server HOST-xx-xx-xx-55/xx.xx.xx.55:2182, sessionid = 0x1379da4660c0014, negotiated timeout = 5000
> 12/05/30 19:48:24 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1379da4660c0014
> 12/05/30 19:48:24 INFO zookeeper.ZooKeeper: Session: 0x1379da4660c0014 closed
> 12/05/30 19:48:24 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/hacluster'
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:263)
> at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:257)
> at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:195)
> at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:58)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:163)
> at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:159)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:438)
> at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:159)
> at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:171)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hadoop-ha/hacluster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
> at org.apache.hadoop.ha.ActiveStandbyElector.parentZNodeExists(ActiveStandbyElector.java:261)
> ... 8 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3477) FormatZK and ZKFC startup can fail due
to zkclient connection establishment delay
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-3477:
--------------------------------------
Component/s: auto-failover
Target Version/s: 2.0.1-alpha, 3.0.0
Affects Version/s: 3.0.0
> FormatZK and ZKFC startup can fail due to zkclient connection establishment delay
> ---------------------------------------------------------------------------------
>
> Key: HDFS-3477
> URL: https://issues.apache.org/jira/browse/HDFS-3477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: auto-failover
> Affects Versions: 3.0.0
> Reporter: suja s
> Assignee: Rakesh R
>
> Format and ZKFC startup flows continue further after creation of zkclient connection without waiting to check whether the connection is completely established. This leads to failure at the subsequent point if connection was not complete by then.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira