You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2007/09/11 07:48:32 UTC
[jira] Created: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
multiple dfs.client.buffer.dir directories are not treated as alternatives
--------------------------------------------------------------------------
Key: HADOOP-1875
URL: https://issues.apache.org/jira/browse/HADOOP-1875
Project: Hadoop
Issue Type: Bug
Reporter: Christian Kunz
Fix For: 0.15.0
When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1875:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Hairong!
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch, localDirAllocator2.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz updated HADOOP-1875:
-----------------------------------
Component/s: mapred
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Christian Kunz
> Fix For: 0.15.0
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533721 ]
Hudson commented on HADOOP-1875:
--------------------------------
Integrated in Hadoop-Nightly #267 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/267/])
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch, localDirAllocator2.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1875:
----------------------------------
Attachment: dirAllocator.patch
I ran several tests and found out that createFile failed in the case that one of the directories specified in dfs.client.buffer.dir does not exist and it is on a readonly disk. The uploaded patch fixes this problem. Christian, could you please help me check if the patch solves your reported problem? Thanks.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1875:
---------------------------------
Component/s: (was: mapred)
fs
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Fix For: 0.15.0
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-1875:
----------------------------------------
Assignee: Hairong Kuang (was: dhruba borthakur)
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526555 ]
Raghu Angadi commented on HADOOP-1875:
--------------------------------------
DFS client uses {{LocalDirAllocator.createTmpFileWrite()}}
{{LocalDirAllocator}} explicitly states that it does not check for write permissions. I think it should try different directory when create on one dir fails (for whatever reason).
Also during initialization it ignores the error while trying to {{mkdirs(dir[i])}}.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Christian Kunz
> Fix For: 0.15.0
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533494 ]
Owen O'Malley commented on HADOOP-1875:
---------------------------------------
It looks good, except that you should always call them exist/mkdir in the other order:
if (fs.mkdir(path) || fs.exists(path)) { ... }
that way if the code is run in multiple threads, it never returns false.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531869 ]
Hairong Kuang commented on HADOOP-1875:
---------------------------------------
By simply reading the code, it seems that LocalDirAllocator does fail over to a different directory when a tmp directory is not writable when allocating a new directory. But if the allocated directory becomes not writable while data is writing, dfs does not handle the error.
Christian, could you please publish the failure stack trace? So I am able to pinpoint the error. Thanks.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1875:
----------------------------------
Attachment: localDirAllocator1.patch
This patch includes test cases.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1875:
----------------------------------
Status: Patch Available (was: Open)
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch, localDirAllocator2.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1875:
----------------------------------
Assignee: dhruba borthakur
Priority: Blocker (was: Major)
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.15.0
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1875:
----------------------------------
Attachment: localDirAllocator2.patch
Thank you Owen for reviewing the patch. This new patch reflects his comment.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch, localDirAllocator2.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1875) multiple dfs.client.buffer.dir
directories are not treated as alternatives
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533545 ]
Hadoop QA commented on HADOOP-1875:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367402/localDirAllocator2.patch
against trunk revision r583309.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/912/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/912/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/912/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/912/console
This message is automatically generated.
> multiple dfs.client.buffer.dir directories are not treated as alternatives
> --------------------------------------------------------------------------
>
> Key: HADOOP-1875
> URL: https://issues.apache.org/jira/browse/HADOOP-1875
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Reporter: Christian Kunz
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: dirAllocator.patch, localDirAllocator1.patch, localDirAllocator2.patch
>
>
> When specifying multiple directories in the value for dfs.client.buffer.dir, jobs fail when the selected directory does not exist or is not writable. Correct behaviour should be to create the directory when it does not exist and fail over to an alternative directory when it is not writable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.