You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Pete Wyckoff (JIRA)" <ji...@apache.org> on 2008/06/03 20:30:44 UTC
[jira] Created: (HADOOP-3485) implement writes
implement writes
----------------
Key: HADOOP-3485
URL: https://issues.apache.org/jira/browse/HADOOP-3485
Project: Hadoop Core
Issue Type: New Feature
Components: contrib/fuse-dfs
Reporter: Pete Wyckoff
Priority: Minor
Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
Doug, Craig, Nicholas - Comments?
-- pete
ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602045#action_12602045 ]
Doug Cutting commented on HADOOP-3485:
--------------------------------------
This sounds reasonable to me.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Status: Open (was: Patch Available)
uploading a new one that includes check for O_RDWR and allows for partial writes (i.e., does not force #of bytes requested to equal #of actual bytes written.
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: patch1.txt
my first cut at fixing writes and also at cleaning up deletions moving things to trash (when the flag for that is enabled :))
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Status: Open (was: Patch Available)
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Description:
1. dfs_write should return the #of bytes written and not 0
2. implement dfs_flush
3. uncomment/fix dfs_create
4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
was:
Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
Doug, Craig, Nicholas - Comments?
-- pete
ps since mostly already implemented, this should be a very quick patch
Summary: fix writes (was: implement writes)
Fixing the description to more accurately reflect what this issue it about. (now that I know on kernels > 2.6.15, fuse will not try opening and closing and then opening the file if the dfs_create function is there.
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3485:
------------------------------------
Release Note: Introduce write support for Fuse; requires Linux kernel 2.6.15 or better. (was: Introduce write support for Fuse.)
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Affects Versions: 0.19.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt, TEST-TestFuseDFS.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614811#action_12614811 ]
Pete Wyckoff commented on HADOOP-3485:
--------------------------------------
Doug or Dhruba, I think this is ready to be committed as it passes QA and Zheng reviewed it.
thanks, pete
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) fix writes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624790#action_12624790 ]
Hudson commented on HADOOP-3485:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Affects Versions: 0.19.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt, TEST-TestFuseDFS.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614554#action_12614554 ]
Pete Wyckoff commented on HADOOP-3485:
--------------------------------------
I just looked at the code again and see that instead of returning size from the dfs_write, it should return the #of bytes actually written. And instead of forcing those 2 be the same, the error condition should just check that #of bytes written > 0.
I will supply a new patch.
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613751#action_12613751 ]
Hadoop QA commented on HADOOP-3485:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12385996/patch3.txt
against trunk revision 676772.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 26 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2862/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2862/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2862/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2862/console
This message is automatically generated.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: patch3.txt
another try - problem was i svn moved fuse-dfs/test to fuse-dfs/src/test. this time i svn rmed and svn - added
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: (was: patch1.txt)
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: patch2.txt
This is the patch for supporting writes and adding write unit test.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) fix writes
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614601#action_12614601 ]
Zheng Shao commented on HADOOP-3485:
------------------------------------
+1
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: TEST-TestFuseDFS.txt
just for documentation, here's the output of the unit tests.
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt, TEST-TestFuseDFS.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3485:
------------------------------------
Release Note: Introduce write support for Fuse. (was: Write support added, but performance of writes not looked at yet.
I also changed the unit tests to be callable from the top level hadoop build file
Note that on Linux, writes require kernel 2.6.15 or above)
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Affects Versions: 0.19.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt, TEST-TestFuseDFS.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: patch4.txt
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Status: Patch Available (was: Open)
patch3.txt
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Chuang Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602398#action_12602398 ]
Chuang Liu commented on HADOOP-3485:
------------------------------------
Hi:
Is it possible to use struct fuse_file_info * to pass file handler created in fuse 'create' call to the following I/O calls like FUSE "write"? In this way, you do not need close the file in fuse "create()" and reopen it in the following fuse "write()" call.
For example, for a linux 'cp' command, The fuse call sequence is like the following:
static int create (const char *, mode_t, struct fuse_file_info *) ;
static int write (const char *, const char *, size_t, off_t,struct fuse_file_info *)
static int flush(...)
static int release(...)
The data in struct fuse_file_info* is passed along the call sequence. Therefore, we could use it to store hadoop file handler or whatever info needed for following I/O info. Thanks.
Chuang
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611837#action_12611837 ]
Pete Wyckoff commented on HADOOP-3485:
--------------------------------------
Ignore above - I now see that open will only do what we want on kernels >= 2.6.15, which I was not using when running the above test.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604283#action_12604283 ]
Craig Macdonald commented on HADOOP-3485:
-----------------------------------------
Is HADOOP-1700 really so far away that we need to hack around it? There's only two pre-req JIRAs remaining.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Chuang Liu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602796#action_12602796 ]
Chuang Liu commented on HADOOP-3485:
------------------------------------
Could you please clarify the situations when this call sequence happens? Could it be fuse version related? I used FUSE 2.6.
Also, in fuse.h, the description of fuse "create" says "If this method is not implemented or under linux kernel versions earlier than 2.6.15, the mknod() and open() methods will be called instead". It might be interesting to try geting rid of 'create' and moving logic in this function to 'open' such that the file gets closed just once?
Chuang
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) fix writes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614663#action_12614663 ]
Hadoop QA commented on HADOOP-3485:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12386360/patch4.txt
against trunk revision 677781.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 26 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2898/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2898/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2898/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2898/console
This message is automatically generated.
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Attachment: patch1.txt
I don't know what I was thinking before about writes not working. Must have been confused. There was a bug (the write not returning #bytes written) which I have fixed and also enabled create, write, mknod and flush.
I'm including this now, but will wait to officially submit a patch until I have a unit test.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Status: Patch Available (was: Open)
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604293#action_12604293 ]
Hong Tang commented on HADOOP-3485:
-----------------------------------
Why would you consider it a hack? Well, maybe true in the sense that writing FUSE for HDFS is a hack because you will never be able to perform random writes.
In any case, my take is that by not depending on appending, the FUSE-based mounter can be used in current and older versions of HDFS, which benefits users who prefer hold-and-wait when some new things come out.
BTW, I do not believe this should be a major effort, and I am reasonably comfortable to say this because my old company has an internal product very similar to HDFS and we got it working with FUSE flawlessly.
-Hong
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Release Note:
Write support added, but performance of writes not looked at yet.
I also changed the unit tests to be callable from the top level hadoop build file
Note that on Linux, writes require kernel 2.6.15 or above
was:
Write support added, but performance of writes not looked at yet.
I also changed the unit tests to be callable from the top level hadoop build file
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3485:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Pete!
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613754#action_12613754 ]
Pete Wyckoff commented on HADOOP-3485:
--------------------------------------
Can one of you do a code review on the patch and we can then ask Doug or someone to commit this?
thx, pete
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-3485:
---------------------------------
Assignee: Pete Wyckoff
Release Note:
Write support added, but performance of writes not looked at yet.
I also changed the unit tests to be callable from the top level hadoop build file
Status: Patch Available (was: Open)
run the unit tests with:
ant test-contrib -Dfusedfs=1 -Dlibhdfs=1 -Dtestcase=TestFuseDFS
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611808#action_12611808 ]
Pete Wyckoff commented on HADOOP-3485:
--------------------------------------
FUSE/Java FileOutputStream is causing the close to happen right after the create. I don't know which. Here's the output from fuse running in debug mode:
[junit] unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
[junit] INIT: 7.8
[junit] flags=0x00000003
[junit] max_readahead=0x00020000
[junit] INIT: 7.8
[junit] flags=0x00000001
[junit] max_readahead=0x00020000
[junit] max_write=0x00020000
[junit] unique: 1, error: 0 (Success), outsize: 40
[junit] unique: 2, opcode: LOOKUP (1), nodeid: 1, insize: 50
[junit] LOOKUP /hello.txt
[junit] unique: 2, error: -2 (No such file or directory), outsize: 16
[junit] unique: 3, opcode: MKNOD (8), nodeid: 1, insize: 58
[junit] MKNOD /hello.txt
[junit] in dfs_create! NODEID: 2
[junit] dfs_release unique: 3, error: 0 (Success), outsize: 136
[junit] unique: 4, opcode: OPEN (14), nodeid: 2, insize: 48
[junit] unique: 4, error: 0 (Success), outsize: 32
[junit] OPEN[12151664] flags: 0x8401 /hello.txt
[junit] unique: 5, opcode: WRITE (16), nodeid: 2, insize: 75
[junit] WRITE[12151664] 11 bytes to 0
[junit] WRITE[12151664] 11 bytes
[junit] unique: 5, error: 0 (Success), outsize: 24
[junit] unique: 6, opcode: FLUSH (25), nodeid: 2, insize: 64
[junit] FLUSH[12151664]
[junit] unique: 6, error: 0 (Success), outsize: 16
[junit] unique: 7, opcode: RELEASE (18), nodeid: 2, insize: 64
[junit] RELEASE[12151664] flags: 0x8401
And here's the java code creating the file:
// create the file
File file = new File(mpoint, "hello.txt");
FileOutputStream f = new FileOutputStream(file);
String s = "hello world";
f.write(s.getBytes());
f.flush();
f.close();
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613081#action_12613081 ]
Hadoop QA commented on HADOOP-3485:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12385690/patch2.txt
against trunk revision 676069.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 36 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2838/console
This message is automatically generated.
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3485) implement writes
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602497#action_12602497 ]
Pete Wyckoff commented on HADOOP-3485:
--------------------------------------
Good question - the problem is that in some situations, fuse seems to do something more like:
create
release
open
write
flush
release.
So, although we already store the file pointer in the fuse_file_info structure, there's nothing we can do if fuse tells us to close the file.
And note, the fuse_file_info pointer passed into open doesn't include the original fuse_file_info data. So, even not closing it on the first release won't help - unless we cached the file handle ourselves.
If you know how we can get around fuse doing this, it would be very helpful.
-- pete
> implement writes
> ----------------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Reporter: Pete Wyckoff
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.
> One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).
> Doug, Craig, Nicholas - Comments?
> -- pete
> ps since mostly already implemented, this should be a very quick patch
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3485) fix writes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3485:
-------------------------------------
Affects Version/s: 0.19.0
> fix writes
> ----------
>
> Key: HADOOP-3485
> URL: https://issues.apache.org/jira/browse/HADOOP-3485
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fuse-dfs
> Affects Versions: 0.19.0
> Reporter: Pete Wyckoff
> Assignee: Pete Wyckoff
> Priority: Minor
> Attachments: patch1.txt, patch2.txt, patch3.txt, patch4.txt, TEST-TestFuseDFS.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> 1. dfs_write should return the #of bytes written and not 0
> 2. implement dfs_flush
> 3. uncomment/fix dfs_create
> 4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
> 5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.