You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Pieter Reuse (JIRA)" <ji...@apache.org> on 2015/11/09 16:45:11 UTC

[jira] [Updated] (HADOOP-11262) Enable YARN to use S3A

     [ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pieter Reuse updated HADOOP-11262:
----------------------------------
    Attachment: HADOOP-11262-8.patch

Thank you, [~eddyxu] and [~cnauroth] for reviewing this and suggesting improvements. I've uploaded patch version 8, addressing [~eddyxu]'s remarks about coding style, @Override in combination with @Before, @After or @Test and a try-with-recourses.

Regarding the setVerifyChecksum-test, we ([~Thomas Demoor] and I) noticed that the default FileSystem - and therefore S3A - simply ignores the "setVerifyChecksum"-flag, and that the test is therefore unnecessary for S3A. So overriding it with an empty method avoids having a falsely failing test.

The original observation was that this flag is only used in the inputstream, and calling setVerifyChecksum() before out.close() is called makes S3A throw an IOException. Simply changing the order of these two lines fixed the issue we faced there, but patch 8 uses a different approach and does not change the testSetVerifyChecksum()-method in the super class any more.

> Enable YARN to use S3A 
> -----------------------
>
>                 Key: HADOOP-11262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11262
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Thomas Demoor
>            Assignee: Pieter Reuse
>              Labels: amazon, s3
>         Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch
>
>
> Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)