You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org> on 2013/11/19 20:43:23 UTC
[jira] [Comment Edited] (HDFS-5014)
BPOfferService#processCommandFromActor() synchronization on namenode RPC
call delays IBR to Active NN, if Stanby NN is unstable
[ https://issues.apache.org/jira/browse/HDFS-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826859#comment-13826859 ]
Uma Maheswara Rao G edited comment on HDFS-5014 at 11/19/13 7:42 PM:
---------------------------------------------------------------------
Seems like there is an issue with the path.
We should move the cmd null check before DNA_REGISTER if condition.
{noformat}
2013-11-19 14:08:33,394 ERROR datanode.DataNode (BPServiceActor.java:run(719)) - Exception in BPOfferService for Block pool BP-1297942247-67.195.138.31-1384870112818 (storage id DS-234026112-67.195.138.31-43443-1384870113355) service to localhost/127.0.0.1:48821
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:507)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:745)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:597)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:717)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This would be the reason for timeouts in Jenkins.
was (Author: umamaheswararao):
Seems like there is an issue with the path.
We should move the cmd null check before DNA_REGISTER if condition.
{noformat}
2013-11-19 14:08:33,394 ERROR datanode.DataNode (BPServiceActor.java:run(719)) - Exception in BPOfferService for Block pool BP-1297942247-67.195.138.31-1384870112818 (storage id DS-234026112-67.195.138.31-43443-1384870113355) service to localhost/127.0.0.1:48821
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:507)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:745)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:597)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:717)
at java.lang.Thread.run(Thread.java:662)
{code}
> BPOfferService#processCommandFromActor() synchronization on namenode RPC call delays IBR to Active NN, if Stanby NN is unstable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-5014
> URL: https://issues.apache.org/jira/browse/HDFS-5014
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, ha
> Affects Versions: 3.0.0, 2.0.4-alpha
> Reporter: Vinay
> Assignee: Vinay
> Attachments: HDFS-5014-v2.patch, HDFS-5014-v2.patch, HDFS-5014-v2.patch, HDFS-5014-v2.patch, HDFS-5014.patch, HDFS-5014.patch, HDFS-5014.patch, HDFS-5014.patch, HDFS-5014.patch, HDFS-5014.patch, HDFS-5014.patch
>
>
> In one of our cluster, following has happened which failed HDFS write.
> 1. Standby NN was unstable and continously restarting due to some errors. But Active NN was stable.
> 2. MR Job was writing files.
> 3. At some point SNN went down again while datanode processing the REGISTER command for SNN.
> 4. Datanodes started retrying to connect to SNN to register at the following code in BPServiceActor#retrieveNamespaceInfo() which will be called under synchronization.
> {code} try {
> nsInfo = bpNamenode.versionRequest();
> LOG.debug(this + " received versionRequest response: " + nsInfo);
> break;{code}
> Unfortunately in all datanodes at same point this happened.
> 5. For next 7-8 min standby was down, and no blocks were reported to active NN at this point and writes have failed.
> So culprit is {{BPOfferService#processCommandFromActor()}} is completely synchronized which is not required.
--
This message was sent by Atlassian JIRA
(v6.1#6144)