You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2017/09/16 15:35:51 UTC
Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Hi,
Some refactoring of the Celery config is underway and as some of the options have changed between Celery 3 and 4 I have asked the question whether Celery 3 is still supported. Apparently it is not (https://github.com/celery/celery/issues/4258 <https://github.com/celery/celery/issues/4258>).
As Celery 3 is also 2 major releases behind I propose to set Celery 4.0 as a minimum supported version with Celery 4.1 being the recommend version. I think this is also important as we do see some intermittent issues with Celery that are reported to us but are, most likely, issues in Celery. I don’t want to take care of those as they are difficult to debug.
I would like to do this per 1.9.0. The 1.8.X branch can still support Celery 3.
Cheers
Bolke
Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Posted by Ruslan Dautkhanov <da...@gmail.com>.
4.0 minimum
and 4.1 as recommended?
Celery 4.1 re-added sqla transport
https://issues.apache.org/jira/browse/AIRFLOW-797
https://issues.apache.org/jira/browse/AIRFLOW-749
--
Ruslan Dautkhanov
On Tue, Sep 19, 2017 at 9:54 AM, Alex Guziel <alex.guziel@airbnb.com.invalid
> wrote:
> That's probably fine but I'd like to note two things.
>
> 1) The celery 3 config options are forwards compatible as far as I know
> 2) Still doesn't fix the bug where tasks get reserved even though it
> shouldn't.
>
>
> But I think it makes sense to upgrade the version in setup.py regardless.
>
> On Tue, Sep 19, 2017 at 2:39 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > ping.
> >
> > I'd like some more feedback.
> >
> > Cheers
> > Bolke
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 16 sep. 2017 om 17:45 heeft Ash Berlin-Taylor <
> > ash_airflowlist@firemirror.com> het volgende geschreven:
> > >
> > > +1 from us, we're running on Celery 4.0.2 in production on Airflow
> 1.8.2
> > (4.1 wasn't out when we started and haven't upgraded in prod yet)
> > >
> > >
> > >> On 16 Sep 2017, at 16:35, Bolke de Bruin <bd...@gmail.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Some refactoring of the Celery config is underway and as some of the
> > options have changed between Celery 3 and 4 I have asked the question
> > whether Celery 3 is still supported. Apparently it is not (
> > https://github.com/celery/celery/issues/4258 <
> > https://github.com/celery/celery/issues/4258>).
> > >>
> > >> As Celery 3 is also 2 major releases behind I propose to set Celery
> 4.0
> > as a minimum supported version with Celery 4.1 being the recommend
> version.
> > I think this is also important as we do see some intermittent issues with
> > Celery that are reported to us but are, most likely, issues in Celery. I
> > don’t want to take care of those as they are difficult to debug.
> > >>
> > >> I would like to do this per 1.9.0. The 1.8.X branch can still support
> > Celery 3.
> > >>
> > >> Cheers
> > >> Bolke
> > >
> >
>
Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Posted by Alex Guziel <al...@airbnb.com.INVALID>.
That's probably fine but I'd like to note two things.
1) The celery 3 config options are forwards compatible as far as I know
2) Still doesn't fix the bug where tasks get reserved even though it
shouldn't.
But I think it makes sense to upgrade the version in setup.py regardless.
On Tue, Sep 19, 2017 at 2:39 AM Bolke de Bruin <bd...@gmail.com> wrote:
> ping.
>
> I'd like some more feedback.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 16 sep. 2017 om 17:45 heeft Ash Berlin-Taylor <
> ash_airflowlist@firemirror.com> het volgende geschreven:
> >
> > +1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2
> (4.1 wasn't out when we started and haven't upgraded in prod yet)
> >
> >
> >> On 16 Sep 2017, at 16:35, Bolke de Bruin <bd...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> Some refactoring of the Celery config is underway and as some of the
> options have changed between Celery 3 and 4 I have asked the question
> whether Celery 3 is still supported. Apparently it is not (
> https://github.com/celery/celery/issues/4258 <
> https://github.com/celery/celery/issues/4258>).
> >>
> >> As Celery 3 is also 2 major releases behind I propose to set Celery 4.0
> as a minimum supported version with Celery 4.1 being the recommend version.
> I think this is also important as we do see some intermittent issues with
> Celery that are reported to us but are, most likely, issues in Celery. I
> don’t want to take care of those as they are difficult to debug.
> >>
> >> I would like to do this per 1.9.0. The 1.8.X branch can still support
> Celery 3.
> >>
> >> Cheers
> >> Bolke
> >
>
Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Posted by Bolke de Bruin <bd...@gmail.com>.
ping.
I'd like some more feedback.
Cheers
Bolke
Verstuurd vanaf mijn iPad
> Op 16 sep. 2017 om 17:45 heeft Ash Berlin-Taylor <as...@firemirror.com> het volgende geschreven:
>
> +1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2 (4.1 wasn't out when we started and haven't upgraded in prod yet)
>
>
>> On 16 Sep 2017, at 16:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Hi,
>>
>> Some refactoring of the Celery config is underway and as some of the options have changed between Celery 3 and 4 I have asked the question whether Celery 3 is still supported. Apparently it is not (https://github.com/celery/celery/issues/4258 <https://github.com/celery/celery/issues/4258>).
>>
>> As Celery 3 is also 2 major releases behind I propose to set Celery 4.0 as a minimum supported version with Celery 4.1 being the recommend version. I think this is also important as we do see some intermittent issues with Celery that are reported to us but are, most likely, issues in Celery. I don’t want to take care of those as they are difficult to debug.
>>
>> I would like to do this per 1.9.0. The 1.8.X branch can still support Celery 3.
>>
>> Cheers
>> Bolke
>
Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Posted by Bolke de Bruin <bd...@gmail.com>.
4.1 has re-added support. We can default to 4.1 on Travis as well. Redis/rabbitmq can both work on Travis as well indeed.
B.
Sent from my iPhone
> On 19 Sep 2017, at 19:26, Maxime Beauchemin <ma...@gmail.com> wrote:
>
> Quick note that Celery 4 drops support for the SQLAlchemy broker which was
> convenient for running unit tests. We'll have to move to using Redis for
> tests which is trivial on Travis.
>
> On Sat, Sep 16, 2017 at 8:45 AM, Ash Berlin-Taylor <
> ash_airflowlist@firemirror.com> wrote:
>
>> +1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2
>> (4.1 wasn't out when we started and haven't upgraded in prod yet)
>>
>>
>>> On 16 Sep 2017, at 16:35, Bolke de Bruin <bd...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Some refactoring of the Celery config is underway and as some of the
>> options have changed between Celery 3 and 4 I have asked the question
>> whether Celery 3 is still supported. Apparently it is not (
>> https://github.com/celery/celery/issues/4258 <https://github.com/celery/
>> celery/issues/4258>).
>>>
>>> As Celery 3 is also 2 major releases behind I propose to set Celery 4.0
>> as a minimum supported version with Celery 4.1 being the recommend version.
>> I think this is also important as we do see some intermittent issues with
>> Celery that are reported to us but are, most likely, issues in Celery. I
>> don’t want to take care of those as they are difficult to debug.
>>>
>>> I would like to do this per 1.9.0. The 1.8.X branch can still support
>> Celery 3.
>>>
>>> Cheers
>>> Bolke
>>
>>
Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Posted by Maxime Beauchemin <ma...@gmail.com>.
Quick note that Celery 4 drops support for the SQLAlchemy broker which was
convenient for running unit tests. We'll have to move to using Redis for
tests which is trivial on Travis.
On Sat, Sep 16, 2017 at 8:45 AM, Ash Berlin-Taylor <
ash_airflowlist@firemirror.com> wrote:
> +1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2
> (4.1 wasn't out when we started and haven't upgraded in prod yet)
>
>
> > On 16 Sep 2017, at 16:35, Bolke de Bruin <bd...@gmail.com> wrote:
> >
> > Hi,
> >
> > Some refactoring of the Celery config is underway and as some of the
> options have changed between Celery 3 and 4 I have asked the question
> whether Celery 3 is still supported. Apparently it is not (
> https://github.com/celery/celery/issues/4258 <https://github.com/celery/
> celery/issues/4258>).
> >
> > As Celery 3 is also 2 major releases behind I propose to set Celery 4.0
> as a minimum supported version with Celery 4.1 being the recommend version.
> I think this is also important as we do see some intermittent issues with
> Celery that are reported to us but are, most likely, issues in Celery. I
> don’t want to take care of those as they are difficult to debug.
> >
> > I would like to do this per 1.9.0. The 1.8.X branch can still support
> Celery 3.
> >
> > Cheers
> > Bolke
>
>
Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
+1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2 (4.1 wasn't out when we started and haven't upgraded in prod yet)
> On 16 Sep 2017, at 16:35, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Hi,
>
> Some refactoring of the Celery config is underway and as some of the options have changed between Celery 3 and 4 I have asked the question whether Celery 3 is still supported. Apparently it is not (https://github.com/celery/celery/issues/4258 <https://github.com/celery/celery/issues/4258>).
>
> As Celery 3 is also 2 major releases behind I propose to set Celery 4.0 as a minimum supported version with Celery 4.1 being the recommend version. I think this is also important as we do see some intermittent issues with Celery that are reported to us but are, most likely, issues in Celery. I don’t want to take care of those as they are difficult to debug.
>
> I would like to do this per 1.9.0. The 1.8.X branch can still support Celery 3.
>
> Cheers
> Bolke