You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/04/05 00:00:32 UTC
[jira] Created: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
The open method of FSNamesystem should be synchronized
------------------------------------------------------
Key: HADOOP-1205
URL: https://issues.apache.org/jira/browse/HADOOP-1205
Project: Hadoop
Issue Type: Bug
Components: dfs
Affects Versions: 0.12.2
Reporter: Hairong Kuang
Fix For: 0.13.0
Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1205:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Hairong.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495680 ]
dhruba borthakur commented on HADOOP-1205:
------------------------------------------
Maybe we can make the open method synchronized on the FSNamesystem global lock for the 0.13 release. A more finer-grain lockign odel is part of 1309 and will be scheduled for the following release.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1205:
----------------------------------
Assignee: Hairong Kuang
Status: Patch Available (was: Open)
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494420 ]
Raghu Angadi commented on HADOOP-1205:
--------------------------------------
Other modifications to blocksMap don't lock blocksMap. Many places the the access is under global FSNamesystem lock. But not all accesses to blocksMap is under lock. We will need to fix all of them, possibly by moving it under global lock.
I am not sure if the patch fixes any issue. Can you clarify why the change is correct or helps?
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495803 ]
Hadoop QA commented on HADOOP-1205:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12357311/open applied and successfully tested against trunk revision r537939.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/138/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/138/console
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1205:
-------------------------------------
Priority: Blocker (was: Major)
Possible blocker for 0.13 release. Investigation in progress.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495779 ]
dhruba borthakur commented on HADOOP-1205:
------------------------------------------
+1. Code reviewed.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486793 ]
Owen O'Malley commented on HADOOP-1205:
---------------------------------------
Can we remove FSDirectory.waitForReady too? I believe it is just used to wait for the image to be loaded.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494603 ]
Hairong Kuang commented on HADOOP-1205:
---------------------------------------
I did not realize that blocksMap is protected by a global lock. I do not think that this is a good approach to ensure a structure integrity by acquiring a global lock. But anyway I can change my patch to acquire a global lock. But there are some non-public methods in FSNamesystem that do not acquire a global lock but access blocksMap. Shall we fix them?
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495190 ]
Owen O'Malley commented on HADOOP-1205:
---------------------------------------
I think this needs to be fixed for 0.13. I would propose fixing it as Raghu suggests by locking the FSNameSystem for open.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1205:
----------------------------------
Attachment: open
This patch makes both open and getDatanodeHints synchronized on FSNamesystem global lock. Similiar to open, getDatanodeHints also access blocksMap, but was not synchronized.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495681 ]
dhruba borthakur commented on HADOOP-1205:
------------------------------------------
I meant "hadoop-1306" and not (hadoop-1309).
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1205:
----------------------------------
Attachment: (was: open.patch)
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1205:
----------------------------------
Priority: Major (was: Blocker)
I hesitate to add a global lock to open since it will slowdown map/reduce jobs when all tasks open a file in the beginning. I'd like to mark this issue as a non-blocker for two reasons. First I do not think that the chance of getting inconsistent data is high. Even if open may return inconsistent data, dfsClient has a retry mechanism which allows it to recover from any error. Second Dhruba is working on adding fine grain locks to dfs. This issue can be easily resolved after his work is done.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495136 ]
dhruba borthakur commented on HADOOP-1205:
------------------------------------------
I agree with Hairong. I am removing this one from the Blocker list.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1205:
----------------------------------
Attachment: open.patch
This patch makes sure that the access to clusterMap is synchronized in open.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494523 ]
dhruba borthakur commented on HADOOP-1205:
------------------------------------------
I agree with Raghu. In the current code, the blocksmap is protected by the global FSNamesystem lock.
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open.patch
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496259 ]
Hadoop QA commented on HADOOP-1205:
-----------------------------------
Integrated in Hadoop-Nightly #90 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/90/)
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1205) The open method of FSNamesystem
should be synchronized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1205:
----------------------------------
Priority: Blocker (was: Major)
> The open method of FSNamesystem should be synchronized
> ------------------------------------------------------
>
> Key: HADOOP-1205
> URL: https://issues.apache.org/jira/browse/HADOOP-1205
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.2
> Reporter: Hairong Kuang
> Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: open
>
>
> Method open accesses a shared data structure blockMap. If synchronization access is not enforced, open might return inconsistent results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.