You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Alexandre Vermeerbergen <av...@gmail.com> on 2020/03/26 17:14:07 UTC
How about switching from Celery to RQ?
Hi there,
Looks like couple of projects have moved from Celery to RQ:
* https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
* https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
or are planning such move:
* https://github.com/getredash/redash/issues/4092
Is such move considered for Airflow?
Kind regards,
Alexandre
Re: How about switching from Celery to RQ?
Posted by Tomasz Urbaszek <tu...@apache.org>.
I think we once tried to abandon Celery for other stuff (kombu,
knative)... If there's no reasonable advantage of RQ I see no reason
in switching to it. However, rewriting the "Queue worker" can help a
lot :)
Also a few days ago one of Celery maintainers expressed interest in
contributing to Airflow. I think it's something that may help us.
https://github.com/apache/airflow/pull/7754
Tomek
On Thu, Mar 26, 2020 at 6:29 PM Daniel Imberman
<da...@gmail.com> wrote:
>
> I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” situation), but I don’t think we’re super set on Celery. Would you be interested in making an AIP to discuss potential benefits?
> On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen <av...@gmail.com>, wrote:
> > Hi there,
> >
> > Looks like couple of projects have moved from Celery to RQ:
> > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
> > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
> >
> > or are planning such move:
> > * https://github.com/getredash/redash/issues/4092
> >
> > Is such move considered for Airflow?
> >
> > Kind regards,
> > Alexandre
Re: How about switching from Celery to RQ?
Posted by Jarek Potiuk <Ja...@polidea.com>.
Clear! Thanks for the explanation!
On Thu, Mar 26, 2020 at 8:08 PM Alexandre Vermeerbergen <
avermeerbergen@gmail.com> wrote:
> We prefer ActiveMQ over RabbitMQ because we are used it deploy it in
> production at scale, in HA way, with the upgrade stuff and so on.
> Rebuilding all this knowledge with RabbitMQ is an hassle for us...
>
> Le jeu. 26 mars 2020 à 19:30, Jarek Potiuk <Ja...@polidea.com> a
> écrit :
> >
> > Out of curiosity why ActiveMQ rather than RabbitMQ (except that it is an
> > Apache project that is)?
> >
> > J,
> >
> > On Thu, Mar 26, 2020 at 6:40 PM Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > To be honest I found RQ while searching why Airflow isn't currently
> > > supporting ActiveMQ instead of RabbitMQ.
> > >
> > > I found that RabbitMQ come from a dependency of Celery on Kombu; and I
> > > found an incident in Kombu discussing about ActiveMQ support:
> > > https://github.com/celery/kombu/issues/548
> > >
> > > I asked if ActiveMQ support was still planned in Kombu, and to my
> > > surprise I got the hint that some projects have moved from Celery to
> > > RQ.
> > >
> > > Now I realize that RQ means using Redis as the queuing infrastructure,
> > > which IHMO is not a so good idea when one realize that it takes 6
> > > nodes to make Redis HA.
> > >
> > > To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if
> > > there's a way to revive interest in adding ActiveMQ support to Kumbu.
> > > After all, Airflow and ActiveMQ are Apache projects, so it seems to be
> > > fair to look for ActiveMQ support for Airflow to get an Apache-based
> > > stack, isn't it ?
> > >
> > > Kind regards,
> > > Alexandre
> > >
> > > Le jeu. 26 mars 2020 à 18:29, Daniel Imberman
> > > <da...@gmail.com> a écrit :
> > > >
> > > > I mean… we’re not planning it (kind of an “if it’s not broke don’t
> fix
> > > it” situation), but I don’t think we’re super set on Celery. Would you
> be
> > > interested in making an AIP to discuss potential benefits?
> > > > On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen <
> > > avermeerbergen@gmail.com>, wrote:
> > > > > Hi there,
> > > > >
> > > > > Looks like couple of projects have moved from Celery to RQ:
> > > > > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
> > > > > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
> > > > >
> > > > > or are planning such move:
> > > > > * https://github.com/getredash/redash/issues/4092
> > > > >
> > > > > Is such move considered for Airflow?
> > > > >
> > > > > Kind regards,
> > > > > Alexandre
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: How about switching from Celery to RQ?
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
We prefer ActiveMQ over RabbitMQ because we are used it deploy it in
production at scale, in HA way, with the upgrade stuff and so on.
Rebuilding all this knowledge with RabbitMQ is an hassle for us...
Le jeu. 26 mars 2020 à 19:30, Jarek Potiuk <Ja...@polidea.com> a écrit :
>
> Out of curiosity why ActiveMQ rather than RabbitMQ (except that it is an
> Apache project that is)?
>
> J,
>
> On Thu, Mar 26, 2020 at 6:40 PM Alexandre Vermeerbergen <
> avermeerbergen@gmail.com> wrote:
>
> > Hi,
> >
> > To be honest I found RQ while searching why Airflow isn't currently
> > supporting ActiveMQ instead of RabbitMQ.
> >
> > I found that RabbitMQ come from a dependency of Celery on Kombu; and I
> > found an incident in Kombu discussing about ActiveMQ support:
> > https://github.com/celery/kombu/issues/548
> >
> > I asked if ActiveMQ support was still planned in Kombu, and to my
> > surprise I got the hint that some projects have moved from Celery to
> > RQ.
> >
> > Now I realize that RQ means using Redis as the queuing infrastructure,
> > which IHMO is not a so good idea when one realize that it takes 6
> > nodes to make Redis HA.
> >
> > To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if
> > there's a way to revive interest in adding ActiveMQ support to Kumbu.
> > After all, Airflow and ActiveMQ are Apache projects, so it seems to be
> > fair to look for ActiveMQ support for Airflow to get an Apache-based
> > stack, isn't it ?
> >
> > Kind regards,
> > Alexandre
> >
> > Le jeu. 26 mars 2020 à 18:29, Daniel Imberman
> > <da...@gmail.com> a écrit :
> > >
> > > I mean… we’re not planning it (kind of an “if it’s not broke don’t fix
> > it” situation), but I don’t think we’re super set on Celery. Would you be
> > interested in making an AIP to discuss potential benefits?
> > > On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com>, wrote:
> > > > Hi there,
> > > >
> > > > Looks like couple of projects have moved from Celery to RQ:
> > > > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
> > > > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
> > > >
> > > > or are planning such move:
> > > > * https://github.com/getredash/redash/issues/4092
> > > >
> > > > Is such move considered for Airflow?
> > > >
> > > > Kind regards,
> > > > Alexandre
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: How about switching from Celery to RQ?
Posted by Jarek Potiuk <Ja...@polidea.com>.
Out of curiosity why ActiveMQ rather than RabbitMQ (except that it is an
Apache project that is)?
J,
On Thu, Mar 26, 2020 at 6:40 PM Alexandre Vermeerbergen <
avermeerbergen@gmail.com> wrote:
> Hi,
>
> To be honest I found RQ while searching why Airflow isn't currently
> supporting ActiveMQ instead of RabbitMQ.
>
> I found that RabbitMQ come from a dependency of Celery on Kombu; and I
> found an incident in Kombu discussing about ActiveMQ support:
> https://github.com/celery/kombu/issues/548
>
> I asked if ActiveMQ support was still planned in Kombu, and to my
> surprise I got the hint that some projects have moved from Celery to
> RQ.
>
> Now I realize that RQ means using Redis as the queuing infrastructure,
> which IHMO is not a so good idea when one realize that it takes 6
> nodes to make Redis HA.
>
> To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if
> there's a way to revive interest in adding ActiveMQ support to Kumbu.
> After all, Airflow and ActiveMQ are Apache projects, so it seems to be
> fair to look for ActiveMQ support for Airflow to get an Apache-based
> stack, isn't it ?
>
> Kind regards,
> Alexandre
>
> Le jeu. 26 mars 2020 à 18:29, Daniel Imberman
> <da...@gmail.com> a écrit :
> >
> > I mean… we’re not planning it (kind of an “if it’s not broke don’t fix
> it” situation), but I don’t think we’re super set on Celery. Would you be
> interested in making an AIP to discuss potential benefits?
> > On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>, wrote:
> > > Hi there,
> > >
> > > Looks like couple of projects have moved from Celery to RQ:
> > > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
> > > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
> > >
> > > or are planning such move:
> > > * https://github.com/getredash/redash/issues/4092
> > >
> > > Is such move considered for Airflow?
> > >
> > > Kind regards,
> > > Alexandre
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: How about switching from Celery to RQ?
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hi,
To be honest I found RQ while searching why Airflow isn't currently
supporting ActiveMQ instead of RabbitMQ.
I found that RabbitMQ come from a dependency of Celery on Kombu; and I
found an incident in Kombu discussing about ActiveMQ support:
https://github.com/celery/kombu/issues/548
I asked if ActiveMQ support was still planned in Kombu, and to my
surprise I got the hint that some projects have moved from Celery to
RQ.
Now I realize that RQ means using Redis as the queuing infrastructure,
which IHMO is not a so good idea when one realize that it takes 6
nodes to make Redis HA.
To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if
there's a way to revive interest in adding ActiveMQ support to Kumbu.
After all, Airflow and ActiveMQ are Apache projects, so it seems to be
fair to look for ActiveMQ support for Airflow to get an Apache-based
stack, isn't it ?
Kind regards,
Alexandre
Le jeu. 26 mars 2020 à 18:29, Daniel Imberman
<da...@gmail.com> a écrit :
>
> I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” situation), but I don’t think we’re super set on Celery. Would you be interested in making an AIP to discuss potential benefits?
> On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen <av...@gmail.com>, wrote:
> > Hi there,
> >
> > Looks like couple of projects have moved from Celery to RQ:
> > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
> > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
> >
> > or are planning such move:
> > * https://github.com/getredash/redash/issues/4092
> >
> > Is such move considered for Airflow?
> >
> > Kind regards,
> > Alexandre
Re: How about switching from Celery to RQ?
Posted by Daniel Imberman <da...@gmail.com>.
I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” situation), but I don’t think we’re super set on Celery. Would you be interested in making an AIP to discuss potential benefits?
On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen <av...@gmail.com>, wrote:
> Hi there,
>
> Looks like couple of projects have moved from Celery to RQ:
> * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning
> * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq
>
> or are planning such move:
> * https://github.com/getredash/redash/issues/4092
>
> Is such move considered for Airflow?
>
> Kind regards,
> Alexandre