[jira] [Created] (AIRFLOW-1979) Redis celery backend not work on 1.9.0 (configuration is ignored)

["Norio Akagi (JIRA)" <ji...@apache.org>]

Norio Akagi created AIRFLOW-1979:
------------------------------------

             Summary: Redis celery backend not work on 1.9.0 (configuration is ignored)
                 Key: AIRFLOW-1979
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1979
             Project: Apache Airflow
          Issue Type: Bug
          Components: celery, worker
    Affects Versions: 1.9.0
            Reporter: Norio Akagi


Worker tries to connect to RabbigMQ based on a default setting and shows an error as below:


{noformat}
[2018-01-09 16:45:42,778] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2018-01-09 16:45:42,802] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2018-01-09 16:45:43,051] {configuration.py:206} WARNING - section/key [celery/celery_ssl_active] not found in config
[2018-01-09 16:45:43,051] {default_celery.py:41} WARNING - Celery Executor will run without SSL
[2018-01-09 16:45:43,052] {__init__.py:45} INFO - Using executor CeleryExecutor
[2018-01-09 16:45:43,140: WARNING/MainProcess] /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
[2018-01-09 16:45:43,240: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds...
{noformat}

I deploy Airflow on kubernetes so each component (web, scheduler, worker, and flower) is containerized and distributed among nodes. I set {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND}}
 and {{AIRFLOW__CELERY__BROKER_URL}} in environment variables and it can be seen when I run {{printenv}} in a container, but it looks completely ignored.

Moving these values to {{airflow.cfg}} doesn't work either.
It worked just perfectly 1.8 and suddenly stopped working when I upgraded Airflow to 1.9.

Do you have any idea what may cause this configuration issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)