You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jason Chen <ch...@gmail.com> on 2017/01/28 22:09:47 UTC
Airflow + Celery + SQS
Hi Airflow team,
Celery 4 supports AWS SQS
http://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html
We are using Airflow 1.7.1.3
Is there any problem, if we change config to use SQS for CeleryExecutor ?
Thanks.
Jason
Re: Airflow + Celery + SQS
Posted by Jason Chen <ch...@gmail.com>.
Joseph,
Thanks for sharing the ideas and I am looking forward to your announcement
and results.
We do use AWS ElasticCache for Celery and lean on SQS mainly due to the
cost!
-Jason
On Mon, Jan 30, 2017 at 11:01 AM, Joseph Napolitano <
joseph.napolitano@blueapron.com.invalid> wrote:
> Jason,
>
> We've considered SQS as the messaging backend for Celery, mostly due to its
> serverless nature. The disadvantage is that we'd depend on AWS for local
> development.
>
> Given this, we're leaning on Redis for messaging because we can use AWS's
> ElasticCache help manage it. This gives us the reliability of AWS while
> not depending on it during development. Redis can also be used as the
> result store for Celery, but since we're already using postgres/RDS for the
> metadata db, I assume we can have a separate schema for Celery's result
> store in the same database instance.
>
> What we don't want to do is have use different containers (using docker) in
> both dev and production, i.e. relying on Redis or Rabbit locally, while
> using SQS in production.
>
> We're still investing this heavily, but would love to keep a tight loop on
> the subject. If you're in NYC, we're hosting an Airflow meetup in 2 day to
> discuss some of our upcoming implementation ideas. I'm going to reannouce
> it later on mailing list.
>
> Cheers
>
> On Mon, Jan 30, 2017 at 9:59 AM, Jeremiah Lowin <jl...@apache.org> wrote:
>
> > Jason,
> >
> > I don't believe Airflow cares about Celery's backend as long as the task
> > API remains the same. You should be OK (though I haven't tested to
> > confirm).
> >
> > J
> >
> > On Sat, Jan 28, 2017 at 5:09 PM Jason Chen <ch...@gmail.com>
> > wrote:
> >
> > > Hi Airflow team,
> > >
> > > Celery 4 supports AWS SQS
> > > http://docs.celeryproject.org/en/latest/getting-started/
> > brokers/sqs.html
> > >
> > > We are using Airflow 1.7.1.3
> > > Is there any problem, if we change config to use SQS for
> CeleryExecutor ?
> > >
> > > Thanks.
> > >
> > > Jason
> > >
> >
>
>
>
> --
> *Joe Napolitano *| Sr. Data Engineer
> www.blueapron.com | 5 Crosby Street, New York, NY 10013
>
Re: Airflow + Celery + SQS
Posted by Joseph Napolitano <jo...@blueapron.com.INVALID>.
Jason,
We've considered SQS as the messaging backend for Celery, mostly due to its
serverless nature. The disadvantage is that we'd depend on AWS for local
development.
Given this, we're leaning on Redis for messaging because we can use AWS's
ElasticCache help manage it. This gives us the reliability of AWS while
not depending on it during development. Redis can also be used as the
result store for Celery, but since we're already using postgres/RDS for the
metadata db, I assume we can have a separate schema for Celery's result
store in the same database instance.
What we don't want to do is have use different containers (using docker) in
both dev and production, i.e. relying on Redis or Rabbit locally, while
using SQS in production.
We're still investing this heavily, but would love to keep a tight loop on
the subject. If you're in NYC, we're hosting an Airflow meetup in 2 day to
discuss some of our upcoming implementation ideas. I'm going to reannouce
it later on mailing list.
Cheers
On Mon, Jan 30, 2017 at 9:59 AM, Jeremiah Lowin <jl...@apache.org> wrote:
> Jason,
>
> I don't believe Airflow cares about Celery's backend as long as the task
> API remains the same. You should be OK (though I haven't tested to
> confirm).
>
> J
>
> On Sat, Jan 28, 2017 at 5:09 PM Jason Chen <ch...@gmail.com>
> wrote:
>
> > Hi Airflow team,
> >
> > Celery 4 supports AWS SQS
> > http://docs.celeryproject.org/en/latest/getting-started/
> brokers/sqs.html
> >
> > We are using Airflow 1.7.1.3
> > Is there any problem, if we change config to use SQS for CeleryExecutor ?
> >
> > Thanks.
> >
> > Jason
> >
>
--
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013
Re: Airflow + Celery + SQS
Posted by Jeremiah Lowin <jl...@apache.org>.
Jason,
I don't believe Airflow cares about Celery's backend as long as the task
API remains the same. You should be OK (though I haven't tested to
confirm).
J
On Sat, Jan 28, 2017 at 5:09 PM Jason Chen <ch...@gmail.com>
wrote:
> Hi Airflow team,
>
> Celery 4 supports AWS SQS
> http://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html
>
> We are using Airflow 1.7.1.3
> Is there any problem, if we change config to use SQS for CeleryExecutor ?
>
> Thanks.
>
> Jason
>