Falcon and S3

[Sayat Satybaldiyev <sa...@gmail.com>]

Hi,

I'm a new to Falcon project, but I'm thinking to use it for workflow
management tool. I'm wondering if Falcon integrates with S3 well(i.e.
export/import data)?

-- 
Best Regards,
Sayat Satybaldiyev

RE: Falcon and S3

[Srikanth Sundarrajan <sr...@hotmail.com>]

Yes, Falcon can be used to import / export from s3 and then subsequent processesing can be orchestrated by falcon.

Ex: feed definition for s3 export/import (This ex: does an export to s3, by switching the order an import can be realized)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="some-feed-in-s3" description="some-feed-in-s3" xmlns="uri:falcon:feed:0.1">
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(6)"/>
    <clusters>
        <cluster name="my-hdfs-cluster" type="source">
            <validity start="2001-01-00T00:00Z" end="2999-12-31T00:00Z"/>
            <retention limit="days(30)" action="delete"/>
        </cluster>
        <cluster name="my-s3-mirror" type="target">
            <validity start="2001-01-00T00:00Z" end="2999-12-31T00:00Z"/>
            <retention limit="days(90000)" action="delete"/>
            <locations>
                <location type="data" path="s4://my-s3-bucket-name/feed-in-s3/${YEAR}-${MONTH}-${DAY}-${HOUR}/"/>
            </locations>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/my/hdfs/path/feed-in-s3/${YEAR}/${MONTH}/${DAY}/${HOUR}/"/>
    </locations>
    <ACL owner="my-user" group="group" permission="0x755"/>
    <schema location="/schema/mydata" provider="protobuf"/>
</feed>

To avoid exposing the S3 credentials in the s3-url, you can use the following FS client and host the credentials in the user's home directory on hdfs.

https://github.com/InMobi/secure-s3-fsclient

Regards
Srikanth Sundarrajan

From: sayatez@gmail.com
Date: Mon, 20 Jun 2016 11:07:22 +0200
Subject: Falcon and S3
To: user@falcon.apache.org

Hi,
I'm a new to Falcon project, but I'm thinking to use it for workflow management tool. I'm wondering if Falcon integrates with S3 well(i.e. export/import data)?
-- 
Best Regards,
Sayat Satybaldiyev