You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@oozie.apache.org by davideanastasia <gi...@git.apache.org> on 2015/07/08 13:40:13 UTC

[GitHub] oozie pull request: Add support for s3/s3n schemes

GitHub user davideanastasia opened a pull request:

    https://github.com/apache/oozie/pull/13

    Add support for s3/s3n schemes

    The idea is to create a new URIHandler, that uses AWS S3 SDK to check existence of data in S3. Credentials are taken from the IAM Role of the EC2 instance (hence no need of configuration), but I am planning of extending to support custom AWS credential in the oozie-core.xml file.
    
    This patch is particularly useful with Amazon Elastic MapReduce (EMR), because no credentials are stored in the core-site.xml in that case (as EMR will use the custom EmrFileSystem to access S3).
    
    I am a bit unsure about the need of the S3LauncherURIHandler, so I made the methods throw and UnsupportedOperationException, but I would be happy to fill the gap here as well.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/davideanastasia/oozie OOZIE-426

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/oozie/pull/13.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13
    
----
commit 622dcb96505931cfff8c5eabd7b50adeebbbc853
Author: Davide Anastasia <da...@gmail.com>
Date:   2015-07-08T10:45:28Z

    Add S3URIHandler to check job dependencies

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---