You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2008/04/14 21:59:05 UTC
[jira] Created: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Extend FileSystem API to allow appending to files
-------------------------------------------------
Key: HADOOP-3250
URL: https://issues.apache.org/jira/browse/HADOOP-3250
Project: Hadoop Core
Issue Type: New Feature
Components: fs
Reporter: dhruba borthakur
Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-3250:
---------------------------------
Hadoop Flags: [Reviewed]
+1 This looks good to me!
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3250:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.18.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Nicholas
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3250) Extend FileSystem API to
allow appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600972#action_12600972 ]
szetszwo edited comment on HADOOP-3250 at 5/29/08 5:05 PM:
-------------------------------------------------------------------------
3250_20080529.patch:
- Added append API to FileSystem
- Implemented RawLocalFileSystem
- throw new IOException("Not supported") for the other FileSystem subclasses
- need some tests
was (Author: szetszwo):
3250_20080529.patch: need some tests
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Attachments: 3250_20080529.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------
Assignee: Tsz Wo (Nicholas), SZE
Release Note: Added an optional operation, append, to the FileSystem API and an implementation in LocalFileSystem.
Status: Patch Available (was: Open)
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601462#action_12601462 ]
Hadoop QA commented on HADOOP-3250:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383145/3250_20080530.patch
against trunk revision 661918.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/console
This message is automatically generated.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------
Comment: was deleted
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591073#action_12591073 ]
Doug Cutting commented on HADOOP-3250:
--------------------------------------
I like the name 'append' for this.
{noformat}
public FSDataOutputStream append(Path f) {
return append(f, getDefaultBufferSize(), null);
}
public FSDataOutputStream append(Path f, int bufferSize) {
return append(f, buffersize, null);
}
public abstract FSDataOutputStream append(Path f, int bufferSize, Progressible progress);
{noformat}
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637271#action_12637271 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
------------------------------------------------
> The reason that I did not propose adding truncate() to FileSystem.java is that I did not make dramatic change to FileSystem APIs.
There will be a lot of changes. Are we going to apply all these changes to 0.18 and onwards?
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3250:
------------------------------------
Release Note: (was: Added an optional operation, append, to the FileSystem API and an implementation in LocalFileSystem.)
Removed release note so as to not confuse folks about whether 1700 is done.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591068#action_12591068 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
------------------------------------------------
- Literally, the word "create" does not mean or include append operation.
- It seems to me that permission, overwrite, replication and blockSize do not make sense in append. This is the current create header.
{code}
public abstract FSDataOutputStream create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress) throws IOException;
{code}
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------
Attachment: 3250_20080530.patch
3250_20080530.patch:
- implemented FTPFileSystem
- added tests
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------
Attachment: 3250_20080529.patch
3250_20080529.patch: need some tests
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Attachments: 3250_20080529.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591060#action_12591060 ]
dhruba borthakur commented on HADOOP-3250:
------------------------------------------
My mistake. I wanted to that we should create another method called FileSystem.create() with an "append" flag.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588756#action_12588756 ]
dhruba borthakur commented on HADOOP-3250:
------------------------------------------
One option for this API would be a new method in the FileSystem.open() that takes in the pathname to be opened (and permissions, etc) and returns an FSDataOutputStream. The next write into this stream will occur at the end of the file.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591116#action_12591116 ]
dhruba borthakur commented on HADOOP-3250:
------------------------------------------
+1 on Doug's proposal.
If the path exists and is not being written to, then append() will return a OutputStream to the file. If the path does not exist, then append will generate a IOException.
The DFSClient will issue ClientProtocol.append() call to the namenode. If the file was being written to by another writer but it has exceeded the soft-lease period, then the namenode would *start* a distributed lease recovery. However, it will not make the client wait for the recovery to complete. The namenode will return an AlreadyBeingCreatedException as if the soft-limit has not yet expired.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601188#action_12601188 ]
Doug Cutting commented on HADOOP-3250:
--------------------------------------
I think you need to add an implementation for FTPFileSystem now too. Otherwise, this looks good to me.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Attachments: 3250_20080529.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601928#action_12601928 ]
Hudson commented on HADOOP-3250:
--------------------------------
Integrated in Hadoop-trunk #511 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/511/])
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3250_20080529.patch, 3250_20080530.patch
>
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3250) Extend FileSystem API to allow
appending to files
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591055#action_12591055 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
------------------------------------------------
FileSystem.open currently returns a FSDataInputStream. Clearly, we cannot use an InputStream to do append. I guess we might want to define a new method, say "append", in the FileSystem API.
> Extend FileSystem API to allow appending to files
> -------------------------------------------------
>
> Key: HADOOP-3250
> URL: https://issues.apache.org/jira/browse/HADOOP-3250
> Project: Hadoop Core
> Issue Type: New Feature
> Components: fs
> Reporter: dhruba borthakur
>
> Provide an API to allow applications to append data to pre-existing files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.