You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Oleg Aleshko (JIRA)" <ji...@apache.org> on 2010/08/16 16:34:17 UTC
[jira] Created: (HADOOP-6916) Implement append operation for
S3FileSystem
Implement append operation for S3FileSystem
-------------------------------------------
Key: HADOOP-6916
URL: https://issues.apache.org/jira/browse/HADOOP-6916
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 0.20.2
Reporter: Oleg Aleshko
Priority: Minor
Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Aleshko updated HADOOP-6916:
---------------------------------
Attachment: s3_append1.patch
attached patch, code review needed
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.20.2
> Reporter: Oleg Aleshko
> Priority: Minor
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918731#action_12918731 ]
Eli Collins commented on HADOOP-6916:
-------------------------------------
Forgot to mention this could use some additional tests, eg appending to a file already open for append, appending more than a block, etc. See the hdfs append tests for additional ideas.
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 0.20.2, 0.22.0
> Reporter: Oleg Aleshko
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-6916:
--------------------------------
Affects Version/s: 0.22.0
Fix Version/s: 0.22.0
Issue Type: New Feature (was: Bug)
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 0.20.2, 0.22.0
> Reporter: Oleg Aleshko
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Aleshko updated HADOOP-6916:
---------------------------------
Status: Open (was: Patch Available)
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.20.2
> Reporter: Oleg Aleshko
> Priority: Minor
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Aleshko updated HADOOP-6916:
---------------------------------
Status: Patch Available (was: Open)
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.20.2
> Reporter: Oleg Aleshko
> Priority: Minor
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918729#action_12918729 ]
Eli Collins commented on HADOOP-6916:
-------------------------------------
Hey Oleg,
* Doesn't pos and filePos need to be updated when opening for append?
* Doesn't S3OutputStream#setBlocks need to be synchronized? Why does it copy the blocks?
* Make sure the unit tests exercise calling close after append.
* Have you tested this on S3? eg by setting test.fs.s3.name in src/test/core-site.xml?
Hudson currently running, please use ant test-patch from an svn checkout.
Thanks,
Eli
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 0.20.2, 0.22.0
> Reporter: Oleg Aleshko
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6916:
------------------------------
Status: Open (was: Patch Available)
Marking as open while feedback is addressed.
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 0.20.2, 0.22.0
> Reporter: Oleg Aleshko
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912460#action_12912460 ]
Oleg Aleshko commented on HADOOP-6916:
--------------------------------------
Can this be integrated into the upcoming release?
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.20.2
> Reporter: Oleg Aleshko
> Priority: Minor
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6916) Implement append operation for
S3FileSystem
Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Aleshko updated HADOOP-6916:
---------------------------------
Status: Patch Available (was: Open)
> Implement append operation for S3FileSystem
> -------------------------------------------
>
> Key: HADOOP-6916
> URL: https://issues.apache.org/jira/browse/HADOOP-6916
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.20.2
> Reporter: Oleg Aleshko
> Priority: Minor
> Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.