You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Andrii Kinash <an...@adsquare.com> on 2018/01/17 10:51:49 UTC
Airflow 1.8 CeleryExecutor: multiple servers in broker_url
Dear Airflow Community,
I was able to successfully setup my Celery executor pointing to one RabbitMQ server (broker). However when I’m trying to setup it to point to multiple RabbitMQ servers for HA – it doesn’t seem to work, even thought I’m doing it exactly as it’s written in the Celery documentation: http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_url <http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_url>
When I’m using:
> broker_url = [
> 'pyamqp://usr:pwd@host1:5672//airflow1',
> 'pyamqp://usr:pwd@host2:5672//airflow1'
> ]
I’m getting:
> Jan 17 10:19:58 apph05 airflow[14606]: ConfigParser.ParsingError: File contains parsing errors: /…./airflow.cfg
> Jan 17 10:19:58 apph05 airflow[14606]: [line 162]: ']\n’
and when
> broker_url = 'pyamqp://usr:pwd@host1:5672//airflow1;pyamqp://usr:pwd@host2:5672//airflow1
getting:
> Jan 17 10:22:44 apph05 airflow[15336]: File “/…./airflow/usr/local/lib/python2.7/dist-packages/kombu/transport/__init__.py", line 62, in resolve_transport
> Jan 17 10:22:44 apph05 airflow[15336]: raise KeyError('No such transport: {0}'.format(transport))
Also I tried different approaches, with or without quotes etc., but the end result was the same. I’m not really sure whether it’s a bug or am I doing something wrong?
Thank you,
Andrii
Re: Airflow 1.8 CeleryExecutor: multiple servers in broker_url
Posted by Andrii Kinash <an...@adsquare.com>.
Hi Fokko,
My Celery version is 4.0.2
Thank you for the tip regarding upgrade.
Cheers,
Andrii
> On 17. Jan 2018, at 12:34, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Hi Andrii,
>
> When looking in the Airflow code, I'd say this should workd:
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/executors/celery_executor.py#L41
>
> Which version of celery are you using?
>
> Please note, when you are updating to 1.9.1, the config of Celery changes
> to make it more transparent. The Airflow config is then congruent with the
> Celery config:
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#airflow-191
>
> Cheers, Fokko
>
>
>
>
> 2018-01-17 11:51 GMT+01:00 Andrii Kinash <an...@adsquare.com>:
>
>> Dear Airflow Community,
>>
>> I was able to successfully setup my Celery executor pointing to one
>> RabbitMQ server (broker). However when I’m trying to setup it to point to
>> multiple RabbitMQ servers for HA – it doesn’t seem to work, even thought
>> I’m doing it exactly as it’s written in the Celery documentation:
>> http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:
>> setting-broker_url <http://docs.celeryproject.org/en/latest/userguide/
>> configuration.html#std:setting-broker_url>
>>
>>
>> When I’m using:
>>
>>> broker_url = [
>>> 'pyamqp://usr:pwd@host1:5672//airflow1',
>>> 'pyamqp://usr:pwd@host2:5672//airflow1'
>>> ]
>>
>>
>> I’m getting:
>>
>>> Jan 17 10:19:58 apph05 airflow[14606]: ConfigParser.ParsingError: File
>> contains parsing errors: /…./airflow.cfg
>>> Jan 17 10:19:58 apph05 airflow[14606]: [line 162]: ']\n’
>>
>>
>> and when
>>
>>> broker_url = 'pyamqp://usr:pwd@host1:5672//airflow1;pyamqp://usr:pwd@
>> host2:5672//airflow1
>>
>> getting:
>>
>>> Jan 17 10:22:44 apph05 airflow[15336]: File “/…./airflow/usr/local/lib/
>> python2.7/dist-packages/kombu/transport/__init__.py", line 62, in
>> resolve_transport
>>> Jan 17 10:22:44 apph05 airflow[15336]: raise KeyError('No such
>> transport: {0}'.format(transport))
>>
>>
>> Also I tried different approaches, with or without quotes etc., but the
>> end result was the same. I’m not really sure whether it’s a bug or am I
>> doing something wrong?
>>
>> Thank you,
>> Andrii
Re: Airflow 1.8 CeleryExecutor: multiple servers in broker_url
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Andrii,
When looking in the Airflow code, I'd say this should workd:
https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/executors/celery_executor.py#L41
Which version of celery are you using?
Please note, when you are updating to 1.9.1, the config of Celery changes
to make it more transparent. The Airflow config is then congruent with the
Celery config:
https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#airflow-191
Cheers, Fokko
2018-01-17 11:51 GMT+01:00 Andrii Kinash <an...@adsquare.com>:
> Dear Airflow Community,
>
> I was able to successfully setup my Celery executor pointing to one
> RabbitMQ server (broker). However when I’m trying to setup it to point to
> multiple RabbitMQ servers for HA – it doesn’t seem to work, even thought
> I’m doing it exactly as it’s written in the Celery documentation:
> http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:
> setting-broker_url <http://docs.celeryproject.org/en/latest/userguide/
> configuration.html#std:setting-broker_url>
>
>
> When I’m using:
>
> > broker_url = [
> > 'pyamqp://usr:pwd@host1:5672//airflow1',
> > 'pyamqp://usr:pwd@host2:5672//airflow1'
> > ]
>
>
> I’m getting:
>
> > Jan 17 10:19:58 apph05 airflow[14606]: ConfigParser.ParsingError: File
> contains parsing errors: /…./airflow.cfg
> > Jan 17 10:19:58 apph05 airflow[14606]: [line 162]: ']\n’
>
>
> and when
>
> > broker_url = 'pyamqp://usr:pwd@host1:5672//airflow1;pyamqp://usr:pwd@
> host2:5672//airflow1
>
> getting:
>
> > Jan 17 10:22:44 apph05 airflow[15336]: File “/…./airflow/usr/local/lib/
> python2.7/dist-packages/kombu/transport/__init__.py", line 62, in
> resolve_transport
> > Jan 17 10:22:44 apph05 airflow[15336]: raise KeyError('No such
> transport: {0}'.format(transport))
>
>
> Also I tried different approaches, with or without quotes etc., but the
> end result was the same. I’m not really sure whether it’s a bug or am I
> doing something wrong?
>
> Thank you,
> Andrii