You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Austin Bennett <wh...@gmail.com> on 2019/05/05 01:23:20 UTC
Operator for syncing s3 buckets (and recursively copying)?
Hi Airflow Devs,
I am not having any luck finding some S3 functionality that I would like to
use.
Specifically the ability to sync buckets. And, the ability to recursively
copy.
`aws s3 sync s3://source-bucket/ s3://destination-bucket`
or
`aws s3 cp s3://source-bucket/folder/ s3://destination-bucket/folder
--recursive`
It seems I could run these with the bash operator, but I am wondering about
using a python based operator that would likely call boto.
Does anything like this exist and I'm unable to find it? Otherwise, I'd be
happy to work on this if the community felt it would be useful.
Please advise -
Thanks,
Austin
Re: Operator for syncing s3 buckets (and recursively copying)?
Posted by Austin Bennett <wh...@gmail.com>.
Thanks, airflowuser,
Would love to get feedback on AIRFLOW-4736
<https://issues.apache.org/jira/browse/AIRFLOW-4736> and AIRFLOW-473
<https://issues.apache.org/jira/browse/AIRFLOW-4736>5. Specifically,
anything that anyone would suggest as conventions/issues ahead of
submission. I haven't found documentation, but potentially there are
conventions or ideals behind what constitutes an operator (how much
functionality to bake into each one, vs when should be separated into a new
one, etc).
Cheers,
Austin
On Wed, May 22, 2019 at 10:33 PM airflowuser
<ai...@protonmail.com.invalid> wrote:
> You can open a Jira for that and then submit your PR.
> There is a similar request for GCP:
> https://issues.apache.org/jira/browse/AIRFLOW-2842
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, May 23, 2019 6:32 AM, Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
>
> > Hi All,
> >
> > I need s3 sync functionality, and currently use it via bash. Would love
> > for an operator to exist. I see nothing in Jira. If not here, it looks
> > like that would be the place to add a ticket, get any feedback and then
> > build?
> >
> > Thanks,
> > Austin
> >
> > On Sat, May 4, 2019 at 6:23 PM Austin Bennett
> whatwouldaustindo@gmail.com
> > wrote:
> >
> > > Hi Airflow Devs,
> > > I am not having any luck finding some S3 functionality that I would
> like
> > > to use.
> > > Specifically the ability to sync buckets. And, the ability to
> recursively
> > > copy.
> > > `aws s3 sync s3://source-bucket/ s3://destination-bucket`
> > > or
> > > `aws s3 cp s3://source-bucket/folder/ s3://destination-bucket/folder
> --recursive`
> > > It seems I could run these with the bash operator, but I am wondering
> > > about using a python based operator that would likely call boto.
> > > Does anything like this exist and I'm unable to find it? Otherwise, I'd
> > > be happy to work on this if the community felt it would be useful.
> > > Please advise -
> > > Thanks,
> > > Austin
>
>
>
Re: Operator for syncing s3 buckets (and recursively copying)?
Posted by airflowuser <ai...@protonmail.com.INVALID>.
You can open a Jira for that and then submit your PR.
There is a similar request for GCP: https://issues.apache.org/jira/browse/AIRFLOW-2842
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, May 23, 2019 6:32 AM, Austin Bennett <wh...@gmail.com> wrote:
> Hi All,
>
> I need s3 sync functionality, and currently use it via bash. Would love
> for an operator to exist. I see nothing in Jira. If not here, it looks
> like that would be the place to add a ticket, get any feedback and then
> build?
>
> Thanks,
> Austin
>
> On Sat, May 4, 2019 at 6:23 PM Austin Bennett whatwouldaustindo@gmail.com
> wrote:
>
> > Hi Airflow Devs,
> > I am not having any luck finding some S3 functionality that I would like
> > to use.
> > Specifically the ability to sync buckets. And, the ability to recursively
> > copy.
> > `aws s3 sync s3://source-bucket/ s3://destination-bucket`
> > or
> > `aws s3 cp s3://source-bucket/folder/ s3://destination-bucket/folder --recursive`
> > It seems I could run these with the bash operator, but I am wondering
> > about using a python based operator that would likely call boto.
> > Does anything like this exist and I'm unable to find it? Otherwise, I'd
> > be happy to work on this if the community felt it would be useful.
> > Please advise -
> > Thanks,
> > Austin
Re: Operator for syncing s3 buckets (and recursively copying)?
Posted by Austin Bennett <wh...@gmail.com>.
Hi All,
I need s3 sync functionality, and currently use it via bash. Would love
for an operator to exist. I see nothing in Jira. If not here, it looks
like that would be the place to add a ticket, get any feedback and then
build?
Thanks,
Austin
On Sat, May 4, 2019 at 6:23 PM Austin Bennett <wh...@gmail.com>
wrote:
> Hi Airflow Devs,
>
> I am not having any luck finding some S3 functionality that I would like
> to use.
>
> Specifically the ability to sync buckets. And, the ability to recursively
> copy.
>
> `aws s3 sync s3://source-bucket/ s3://destination-bucket`
>
> or
>
> `aws s3 cp s3://source-bucket/folder/ s3://destination-bucket/folder
> --recursive`
>
> It seems I could run these with the bash operator, but I am wondering
> about using a python based operator that would likely call boto.
>
> Does anything like this exist and I'm unable to find it? Otherwise, I'd
> be happy to work on this if the community felt it would be useful.
>
> Please advise -
>
> Thanks,
> Austin
>
>