You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Craig Rodrigues <cr...@gmail.com> on 2018/05/21 07:20:50 UTC
celery problem, AttributeError: async
Hi,
I used a requiremens.txt file with these three lines:
git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
celery>=4.2.0rc3
kombu>=4.2.0
I did
pip install -r requirements.txt
When I started my worker, I got:
[2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor CeleryExecutor
Traceback (most recent call last):
File "/bin/airflow", line 32, in <module>
args.func(args)
File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in worker
worker.run(**options)
File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in run
**kwargs)
File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, in __init__
self.setup_instance(**self.prepare_args(**kwargs))
File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, in setup_instance
self.should_use_eventloop() if use_eventloop is None
File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, in should_use_eve
self._conninfo.transport.implements.async and
File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, in __getattr__
raise AttributeError(key)
AttributeError: async
I reported this as a bug in celery here:
https://github.com/celery/celery/issues/4747
It turns out that this is a known problem. It looks like in kombu, they
renamed a bunch of uses of "async" to "asynchronous", to not clash with the new
'async' keyword in Python 3.7.
In celery, they are catching up and doing the same rename.
Almost at the same time that I reported this bug, the issue was fixed in the master branch of celery:
https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
Hopefully the versions of kombu and celery will stabilize so that it all works properly with airflow.
--
Craig
Re: celery problem, AttributeError: async
Posted by Craig Rodrigues <cr...@gmail.com>.
Will the branch name be the same (v1-10-test) after you rebase?
Will you change the version number in that branch to 1.10: ( see https://github.com/apache/incubator-airflow/pull/3389 )?
I am heavily testing this branch right now, and want to deploy it to my prod system this Friday.
Thanks.
--
Craig
On 2018/05/21 17:58:12, Bolke de Bruin <bd...@gmail.com> wrote:
> We will rebranch 1.10 from master. Sorry, I have been too busy with normal life to be able to follow up on the release of 1.10.
>
> B.
Re: celery problem, AttributeError: async
Posted by Bolke de Bruin <bd...@gmail.com>.
We will rebranch 1.10 from master. Sorry, I have been too busy with normal life to be able to follow up on the release of 1.10.
B.
> On 21 May 2018, at 19:54, Craig Rodrigues <cr...@gmail.com> wrote:
>
> Kaxil,
>
> Thanks for merging this into master.
> What is the procedure to get this into v1-10-test branch?
> I am heavily testing that branch right now, and want to deploy that branch to my prod system
> this Friday.
>
> --
> Craig
>
> On 2018/05/21 14:52:33, Craig Rodrigues <ro...@crodrigues.org> wrote:
>> I have submitted:
>>
>> https://github.com/apache/incubator-airflow/pull/3388
>>
>> --
>> Craig
>
Re: celery problem, AttributeError: async
Posted by Craig Rodrigues <cr...@gmail.com>.
Kaxil,
Thanks for merging this into master.
What is the procedure to get this into v1-10-test branch?
I am heavily testing that branch right now, and want to deploy that branch to my prod system
this Friday.
--
Craig
On 2018/05/21 14:52:33, Craig Rodrigues <ro...@crodrigues.org> wrote:
> I have submitted:
>
> https://github.com/apache/incubator-airflow/pull/3388
>
> --
> Craig
Re: celery problem, AttributeError: async
Posted by Craig Rodrigues <ro...@crodrigues.org>.
I have submitted:
https://github.com/apache/incubator-airflow/pull/3388
--
Craig
On Mon, May 21, 2018 at 7:00 AM Naik Kaxil <k....@reply.com> wrote:
> Thanks. Please do that.
>
> On 21/05/2018, 14:59, "Craig Rodrigues" <cr...@gmail.com> wrote:
>
> celery 4.1.1 was just released last night which has all the async
> problems fixed:
>
> https://github.com/celery/celery/commits/v4.1.1
>
> I'll test this out, and then submit a PR to bump airflow's celery
> version to 4.1.1
>
> --
> Craig
>
> On 2018/05/21 07:20:50, Craig Rodrigues <cr...@gmail.com> wrote:
> > Hi,
> >
> > I used a requiremens.txt file with these three lines:
> >
> > git+
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> > celery>=4.2.0rc3
> > kombu>=4.2.0
> >
> >
> > I did
> >
> > pip install -r requirements.txt
> >
> > When I started my worker, I got:
> >
> > [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor
> CeleryExecutor
> > Traceback (most recent call last):
> > File "/bin/airflow", line 32, in <module>
> > args.func(args)
> > File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line
> 74, in wrapper
> > return f(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line
> 959, in worker
> > worker.run(**options)
> > File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line
> 257, in run
> > **kwargs)
> > File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 101, in __init__
> > self.setup_instance(**self.prepare_args(**kwargs))
> > File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 124, in setup_instance
> > self.should_use_eventloop() if use_eventloop is None
> > File "/usr/lib/python2.7/site-packages/celery/worker/worker.py",
> line 243, in should_use_eve
> > self._conninfo.transport.implements.async and
> > File "/usr/lib/python2.7/site-packages/kombu/transport/base.py",
> line 125, in __getattr__
> > raise AttributeError(key)
> > AttributeError: async
> >
> >
> >
> > I reported this as a bug in celery here:
> >
> > https://github.com/celery/celery/issues/4747
> >
> > It turns out that this is a known problem. It looks like in kombu,
> they
> > renamed a bunch of uses of "async" to "asynchronous", to not clash
> with the new
> > 'async' keyword in Python 3.7.
> >
> > In celery, they are catching up and doing the same rename.
> > Almost at the same time that I reported this bug, the issue was
> fixed in the master branch of celery:
> >
> >
> https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
> >
> > Hopefully the versions of kombu and celery will stabilize so that it
> all works properly with airflow.
> >
> > --
> > Craig
> >
> >
> >
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.naik@reply.com
> www.reply.com
>
Re: celery problem, AttributeError: async
Posted by Naik Kaxil <k....@reply.com>.
Thanks. Please do that.
On 21/05/2018, 14:59, "Craig Rodrigues" <cr...@gmail.com> wrote:
celery 4.1.1 was just released last night which has all the async problems fixed:
https://github.com/celery/celery/commits/v4.1.1
I'll test this out, and then submit a PR to bump airflow's celery version to 4.1.1
--
Craig
On 2018/05/21 07:20:50, Craig Rodrigues <cr...@gmail.com> wrote:
> Hi,
>
> I used a requiremens.txt file with these three lines:
>
> git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> celery>=4.2.0rc3
> kombu>=4.2.0
>
>
> I did
>
> pip install -r requirements.txt
>
> When I started my worker, I got:
>
> [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor CeleryExecutor
> Traceback (most recent call last):
> File "/bin/airflow", line 32, in <module>
> args.func(args)
> File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
> return f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in worker
> worker.run(**options)
> File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in run
> **kwargs)
> File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, in __init__
> self.setup_instance(**self.prepare_args(**kwargs))
> File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, in setup_instance
> self.should_use_eventloop() if use_eventloop is None
> File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, in should_use_eve
> self._conninfo.transport.implements.async and
> File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, in __getattr__
> raise AttributeError(key)
> AttributeError: async
>
>
>
> I reported this as a bug in celery here:
>
> https://github.com/celery/celery/issues/4747
>
> It turns out that this is a known problem. It looks like in kombu, they
> renamed a bunch of uses of "async" to "asynchronous", to not clash with the new
> 'async' keyword in Python 3.7.
>
> In celery, they are catching up and doing the same rename.
> Almost at the same time that I reported this bug, the issue was fixed in the master branch of celery:
>
> https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
>
> Hopefully the versions of kombu and celery will stabilize so that it all works properly with airflow.
>
> --
> Craig
>
>
>
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com
Re: celery problem, AttributeError: async
Posted by Craig Rodrigues <cr...@gmail.com>.
celery 4.1.1 was just released last night which has all the async problems fixed:
https://github.com/celery/celery/commits/v4.1.1
I'll test this out, and then submit a PR to bump airflow's celery version to 4.1.1
--
Craig
On 2018/05/21 07:20:50, Craig Rodrigues <cr...@gmail.com> wrote:
> Hi,
>
> I used a requiremens.txt file with these three lines:
>
> git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> celery>=4.2.0rc3
> kombu>=4.2.0
>
>
> I did
>
> pip install -r requirements.txt
>
> When I started my worker, I got:
>
> [2018-05-21 06:39:23,980] {__init__.py:48} INFO - Using executor CeleryExecutor
> Traceback (most recent call last):
> File "/bin/airflow", line 32, in <module>
> args.func(args)
> File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
> return f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 959, in worker
> worker.run(**options)
> File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 257, in run
> **kwargs)
> File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 101, in __init__
> self.setup_instance(**self.prepare_args(**kwargs))
> File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 124, in setup_instance
> self.should_use_eventloop() if use_eventloop is None
> File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 243, in should_use_eve
> self._conninfo.transport.implements.async and
> File "/usr/lib/python2.7/site-packages/kombu/transport/base.py", line 125, in __getattr__
> raise AttributeError(key)
> AttributeError: async
>
>
>
> I reported this as a bug in celery here:
>
> https://github.com/celery/celery/issues/4747
>
> It turns out that this is a known problem. It looks like in kombu, they
> renamed a bunch of uses of "async" to "asynchronous", to not clash with the new
> 'async' keyword in Python 3.7.
>
> In celery, they are catching up and doing the same rename.
> Almost at the same time that I reported this bug, the issue was fixed in the master branch of celery:
>
> https://github.com/celery/celery/commit/c8ef7ad60b72a194654c58beb04a1d65cd0435ad
>
> Hopefully the versions of kombu and celery will stabilize so that it all works properly with airflow.
>
> --
> Craig
>
>
>