You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Oleg Aleshko (JIRA)" <ji...@apache.org> on 2010/08/16 16:34:17 UTC

[jira] Created: (HADOOP-6916) Implement append operation for S3FileSystem

Implement append operation for S3FileSystem
-------------------------------------------

                 Key: HADOOP-6916
                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 0.20.2
            Reporter: Oleg Aleshko
            Priority: Minor


Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Attachment: s3_append1.patch

attached patch, code review needed

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918731#action_12918731 ] 

Eli Collins commented on HADOOP-6916:
-------------------------------------

Forgot to mention this could use some additional tests, eg appending to a file already open for append, appending more than a block, etc. See the hdfs append tests for additional ideas.

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6916:
--------------------------------

    Affects Version/s: 0.22.0
        Fix Version/s: 0.22.0
           Issue Type: New Feature  (was: Bug)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Status: Open  (was: Patch Available)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Status: Patch Available  (was: Open)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918729#action_12918729 ] 

Eli Collins commented on HADOOP-6916:
-------------------------------------

Hey Oleg,

* Doesn't pos and filePos need to be updated when opening for append?
* Doesn't S3OutputStream#setBlocks need to be synchronized? Why does it copy the blocks?
* Make sure the unit tests exercise calling close after append.
* Have you tested this on S3? eg by setting test.fs.s3.name in src/test/core-site.xml?

Hudson currently running, please use ant test-patch from an svn checkout.

Thanks,
Eli

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Tom White (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6916:
------------------------------

    Status: Open  (was: Patch Available)

Marking as open while feedback is addressed.

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912460#action_12912460 ] 

Oleg Aleshko commented on HADOOP-6916:
--------------------------------------

Can this be integrated into the upcoming release?


> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6916) Implement append operation for S3FileSystem

Posted by "Oleg Aleshko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Status: Patch Available  (was: Open)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.