You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2015/05/28 18:18:18 UTC

[jira] [Created] (HDFS-8491) DN shutdown race conditions with open xceivers

Daryn Sharp created HDFS-8491:
---------------------------------

             Summary: DN shutdown race conditions with open xceivers
                 Key: HDFS-8491
                 URL: https://issues.apache.org/jira/browse/HDFS-8491
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.6.0
            Reporter: Daryn Sharp


DN shutdowns at least for restarts have many race conditions.  Shutdown is very noisy with exceptions.  The DN notifies writers of the restart, waits 1s and then interrupts the xceiver threads but does not join.  The ipc server is stopped and then the bpos services are stopped.

Xceivers then encounter NPEs in closeBlock because the block no longer exists in the volume map when transient storage is checked.  Just before that, the DN notifies the NN the block was received.  This does not appear to always be true, but rather that the thread was interrupted. They race with bpos shutdown, and luckily appear to lose, to send the block received.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)