You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Pete Wyckoff (JIRA)" <ji...@apache.org> on 2008/10/22 22:39:44 UTC
[jira] Created: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
----------------------------------------------------------------------------
Key: HADOOP-4494
URL: https://issues.apache.org/jira/browse/HADOOP-4494
Project: Hadoop Core
Issue Type: Bug
Components: libhdfs
Reporter: Pete Wyckoff
Assignee: Pete Wyckoff
Fix For: 0.19.1, 0.20.0
libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669513#action_12669513 ]
dhruba borthakur commented on HADOOP-4494:
------------------------------------------
Hi Craig, will it be possible for you to review and test this patch on trunk? Also, I would like to put it in 0.19, unless you think otherwise.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641987#action_12641987 ]
Sanjay Radia commented on HADOOP-4494:
--------------------------------------
Minor doc suggestion:
In the documentation in hdfs.h
+ * @param flags Either O_RDONLY or O_WRONLY[|O_APPEND] for read-only or write-only.
O_WRONLY is for create not for writing to arbitrary part of a file.
Update the comment to indicate that O_WRONLY is for creating, etc.
I just noticed that the test uses O_CREAT but the impl does not even bother checking for this (ie treats O_WRONLY as create):
+++ src/c++/libhdfs/hdfs_test.c (working copy)
+ // CREATE
+ hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
Unfortunately fixing this to match posix will break compatibility of apps, since truck libhdfs did not check for O_CREAT either.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-4494:
---------------------------------
Attachment: HADOOP-4494.txt
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-4494:
---------------------------------
Attachment: HADOOP-4494.txt
added more comments to the flags param and also issue a stderr warning if O_CREAT & O_EXCL are but set.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641971#action_12641971 ]
dhruba borthakur commented on HADOOP-4494:
------------------------------------------
+1. Looks good.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-4494:
---------------------------------
Affects Version/s: 0.20.0
0.19.1
Status: Patch Available (was: Open)
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669181#action_12669181 ]
Craig Macdonald commented on HADOOP-4494:
-----------------------------------------
Anything needed to get this patch in? HADOOP-4508 is now committed.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642035#action_12642035 ]
Pete Wyckoff commented on HADOOP-4494:
--------------------------------------
idea - we deprecate not having the O_CREAT flag by writing out a warning and updating the docs.
The good thing about this kind of deprecation is we can ultimately just return -ENOENT when we are ready - should be easy for users to see they have to fix something.
but, should also look at O_EXCL at the same time ...
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-4494:
---------------------------------
Status: Open (was: Patch Available)
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670965#action_12670965 ]
dhruba borthakur commented on HADOOP-4494:
------------------------------------------
Hi Craig, can you pl review this patch? Thanks. Also, I see some earlier review comments from Sanjay.
Sanjay: do you want to review this patch too?
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642021#action_12642021 ]
Pete Wyckoff commented on HADOOP-4494:
--------------------------------------
bq. I just noticed that the test uses O_CREAT but the impl does not even bother checking for this (ie treats O_WRONLY as create):
Yes, open on writes should i guess be O_WRONLY + one of the following:
1. O_CREAT | O_EXCL - create new file
2. O_CREATE|O_APPEND - create new file if it doesn't exist, else append
But, as you point out hdfsOpenFile has only ever checked for O_WRONLY|O_RDONLY. And #2 cannot be done without underlying FileSystem API.
But, as you point out, if someone does something like `echo hello >> foo` where foo does not exist, libhdfs will do the wrong thing probably.
We need:
{code}
FSDataOutputStream FileSystem.appendIfExistsElseCreate(Path p);
{code}
I can open a JIRA for it. I guess that means anyone using libhdfs (e.g. fuse) would have to do a hacky check if the file exists and decide whether to pass on the append flag, which of course is not transactional.
And of course, let's not even get into passing on O_TRUNCATE :) But, the libhdfs code should probably check for that flag and error out if it comes.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671986#action_12671986 ]
dhruba borthakur commented on HADOOP-4494:
------------------------------------------
Thanks Craig. Please do let me know if this patch looks good to you.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-4494:
---------------------------------
Status: Patch Available (was: Open)
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672595#action_12672595 ]
Craig Macdonald commented on HADOOP-4494:
-----------------------------------------
+1 looks good to me.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671058#action_12671058 ]
Craig Macdonald commented on HADOOP-4494:
-----------------------------------------
Hi Dhruba,
Sorry, I'm currently traveling. I can test for end of next week at earliest.
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4494:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Pete!
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4494) libhdfs does not call
FileSystem.append when O_APPEND passed to hdfsOpenFile
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673968#action_12673968 ]
Hudson commented on HADOOP-4494:
--------------------------------
Integrated in Hadoop-trunk #756 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/])
> libhdfs does not call FileSystem.append when O_APPEND passed to hdfsOpenFile
> ----------------------------------------------------------------------------
>
> Key: HADOOP-4494
> URL: https://issues.apache.org/jira/browse/HADOOP-4494
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Fix For: 0.19.1, 0.20.0
>
> Attachments: HADOOP-4494.txt, HADOOP-4494.txt
>
>
> libhdfs currently calls FileSystem.create when O_APPEND passed in - ie it ignores this flag
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.