You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Tanping Wang (JIRA)" <ji...@apache.org> on 2011/02/24 23:42:38 UTC
[jira] Created: (HDFS-1648) Federation: Properly release
in_use.lock created in BlockPoolStorage
Federation: Properly release in_use.lock created in BlockPoolStorage
---------------------------------------------------------------------
Key: HDFS-1648
URL: https://issues.apache.org/jira/browse/HDFS-1648
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: Federation Branch
Reporter: Tanping Wang
Assignee: Tanping Wang
Fix For: Federation Branch
StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
initialization.
Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
<StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1648) Federation: Only DataStorage must be
locked using in_use.lock and no locks must be associated with
BlockPoolStorage
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas resolved HDFS-1648.
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I committed the patch. Thank you Tanping.
> Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1648
> URL: https://issues.apache.org/jira/browse/HDFS-1648
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: Federation Branch
> Reporter: Tanping Wang
> Assignee: Tanping Wang
> Fix For: Federation Branch
>
> Attachments: HDFS-1648.patch
>
>
> StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
> initialization.
> Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
> <StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
> formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
> in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
> instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
> Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1648) Federation: Properly release
in_use.lock created in BlockPoolStorage
Posted by "Tanping Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999113#comment-12999113 ]
Tanping Wang commented on HDFS-1648:
------------------------------------
test-patch result:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
There are lot of tests failing without this change. I ran one such test TestRestartDFS. It passed with this test.
> Federation: Properly release in_use.lock created in BlockPoolStorage
> ---------------------------------------------------------------------
>
> Key: HDFS-1648
> URL: https://issues.apache.org/jira/browse/HDFS-1648
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: Federation Branch
> Reporter: Tanping Wang
> Assignee: Tanping Wang
> Fix For: Federation Branch
>
> Attachments: HDFS-1648.patch
>
>
> StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
> initialization.
> Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
> <StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
> formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
> in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
> instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
> Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1648) Federation: Only DataStorage must be
locked using in_use.lock and no locks must be associated with
BlockPoolStorage
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999614#comment-12999614 ]
Suresh Srinivas commented on HDFS-1648:
---------------------------------------
+1 for the change.
> Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1648
> URL: https://issues.apache.org/jira/browse/HDFS-1648
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: Federation Branch
> Reporter: Tanping Wang
> Assignee: Tanping Wang
> Fix For: Federation Branch
>
> Attachments: HDFS-1648.patch
>
>
> StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
> initialization.
> Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
> <StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
> formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
> in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
> instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
> Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1648) Federation: Only DataStorage must be
locked using in_use.lock and no locks must be associated with
BlockPoolStorage
Posted by "Tanping Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tanping Wang updated HDFS-1648:
-------------------------------
Summary: Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage (was: Federation: Properly release in_use.lock created in BlockPoolStorage )
> Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1648
> URL: https://issues.apache.org/jira/browse/HDFS-1648
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: Federation Branch
> Reporter: Tanping Wang
> Assignee: Tanping Wang
> Fix For: Federation Branch
>
> Attachments: HDFS-1648.patch
>
>
> StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
> initialization.
> Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
> <StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
> formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
> in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
> instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
> Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1648) Federation: Properly release
in_use.lock created in BlockPoolStorage
Posted by "Tanping Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tanping Wang updated HDFS-1648:
-------------------------------
Attachment: HDFS-1648.patch
> Federation: Properly release in_use.lock created in BlockPoolStorage
> ---------------------------------------------------------------------
>
> Key: HDFS-1648
> URL: https://issues.apache.org/jira/browse/HDFS-1648
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: Federation Branch
> Reporter: Tanping Wang
> Assignee: Tanping Wang
> Fix For: Federation Branch
>
> Attachments: HDFS-1648.patch
>
>
> StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
> initialization.
> Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
> <StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
> formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
> in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
> instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
> Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1648) Federation: Only DataStorage must be
locked using in_use.lock and no locks must be associated with
BlockPoolStorage
Posted by "Tanping Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tanping Wang updated HDFS-1648:
-------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HDFS-1052
> Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1648
> URL: https://issues.apache.org/jira/browse/HDFS-1648
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Federation Branch
> Reporter: Tanping Wang
> Assignee: Tanping Wang
> Fix For: Federation Branch
>
> Attachments: HDFS-1648.patch
>
>
> StorageDirectory locks in_use.lock file, when analyzeStorage() method is called during
> initialization.
> Since BlockPoolStorage uses this method, an in_use.lock file is created and locked under BlockPool directory
> <StorageDirectory>/current/bpid/. This lock is not released when datanode shutsdown. Restarting datanode without
> formatting it results in datanode failure to acquire in_use.lock and hence failure to restart.
> in_use.lock file that currently exists in <StorageDirectory> (at the datanode level) is sufficient to prevent two
> instances of datanodes running on the same node. There is no need for such a lock in block pool directories.
> Change is needed so that BlockPoolStorage can disable locking when it creates it's StorageDirectory.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira