You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Valeriy (JIRA)" <ji...@apache.org> on 2018/12/17 13:30:00 UTC
[jira] [Updated] (AIRFLOW-3532) Apache Airflow > 1.8 don't working
with Celery 4.x and don't working Celery 3.x using other than amqp
transport
[ https://issues.apache.org/jira/browse/AIRFLOW-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Valeriy updated AIRFLOW-3532:
-----------------------------
Description:
I needed Airflow > 1.8 with all the necessary fixes in the cluster configuration.
I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with working DAG's. After some time, all DAG's gone away in queued. After restarting the worker, the problem is solved. I tried for a long time to find solutions to this problem (logs in DEBUG mode showed nothing) and found a number of discussions, one them: [https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
*{color:#d04437}As a result, we conclude that Airflow does not work with Celery 4.x!{color}* The code is not adapted to the Celery 4.x.
I decided to try the Celery 3.x and damn I got an WARNING:
{code:java}
[2018-12-17 15:43:11,136: WARNING/MainProcess] /home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.
The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.
If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
You must only enable the serializers that you will actually use.
warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
-------------- celery@myserver v3.1.26.post2 (Cipater)
---- **** -----
--- * *** * -- Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f0093b86470
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> default exchange=default(direct) key=default
{code}
Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other than amqp. About it already wrote here [http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3CJIRA.13129586.1515519138000.610058.1515519180106@Atlassian.JIRA%3E]
My Airflow config:
{code:java}
[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
broker_url = redis://localhost:6400/0
result_backend = db+postgres://airflow:pass@localhost:5434/airflow
flower_host = 0.0.0.0
flower_url_prefix =
flower_port = 5555
default_queue = default
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
{code}
How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
was:
I needed Airflow > 1.8 with all the necessary fixes in the cluster configuration.
I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with working DAG's. After some time, all DAG's gone away in queued. After restarting the worker, the problem is solved. I tried for a long time to find solutions to this problem (logs in DEBUG mode showed nothing) and found a number of discussions, one them: [https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
*{color:#d04437}As a result, we conclude that Airflow does not work with Celery 4.x!{color}* The code is not adapted to the Celery 4.x.
I decided to try the Celery 3.x and damn I got an WARNING:
{code:java}
[2018-12-17 15:43:11,136: WARNING/MainProcess] /home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.
The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.
If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
You must only enable the serializers that you will actually use.
warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
-------------- celery@youlahdp-driver4.p v3.1.26.post2 (Cipater)
---- **** -----
--- * *** * -- Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f0093b86470
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> default exchange=default(direct) key=default
{code}
Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other than amqp. About it already wrote here http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3CJIRA.13129586.1515519138000.610058.1515519180106@Atlassian.JIRA%3E
My Airflow config:
{code:java}
[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
broker_url = redis://localhost:6400/0
result_backend = db+postgres://airflow:pass@localhost:5434/airflow
flower_host = 0.0.0.0
flower_url_prefix =
flower_port = 5555
default_queue = default
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
{code}
How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
> Apache Airflow > 1.8 don't working with Celery 4.x and don't working Celery 3.x using other than amqp transport
> ---------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3532
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3532
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: 1.9.0, 1.10.0, 1.10.1
> Reporter: Valeriy
> Priority: Major
> Labels: celery
>
> I needed Airflow > 1.8 with all the necessary fixes in the cluster configuration.
> I'm used Aiflow 1.10.0/1.10.1 + Celery 4.2.0/4.2.1 and have problem with working DAG's. After some time, all DAG's gone away in queued. After restarting the worker, the problem is solved. I tried for a long time to find solutions to this problem (logs in DEBUG mode showed nothing) and found a number of discussions, one them: [https://stackoverflow.com/questions/43524457/airflow-tasks-queued-but-not-running]
> *{color:#d04437}As a result, we conclude that Airflow does not work with Celery 4.x!{color}* The code is not adapted to the Celery 4.x.
> I decided to try the Celery 3.x and damn I got an WARNING:
> {code:java}
> [2018-12-17 15:43:11,136: WARNING/MainProcess] /home/hadoop/youla_airflow/lib/python3.6/site-packages/celery/apps/worker.py:161: CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
>
> -------------- celery@myserver v3.1.26.post2 (Cipater)
> ---- **** -----
> --- * *** * -- Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
> -- * - **** ---
> - ** ---------- [config]
> - ** ---------- .> app: airflow.executors.celery_executor:0x7f0093b86470
> - ** ---------- .> transport: amqp://guest:**@localhost:5672//
> - ** ---------- .> results: disabled://
> - *** --- * --- .> concurrency: 16 (prefork)
> -- ******* ----
> --- ***** ----- [queues]
> -------------- .> default exchange=default(direct) key=default
> {code}
> Airflow > 1.8 version with Celery 3.x flatly refuses to use transport other than amqp. About it already wrote here [http://mail-archives.apache.org/mod_mbox/airflow-commits/201801.mbox/%3CJIRA.13129586.1515519138000.610058.1515519180106@Atlassian.JIRA%3E]
> My Airflow config:
> {code:java}
> [celery]
> celery_app_name = airflow.executors.celery_executor
> worker_concurrency = 16
> worker_log_server_port = 8793
> broker_url = redis://localhost:6400/0
> result_backend = db+postgres://airflow:pass@localhost:5434/airflow
> flower_host = 0.0.0.0
> flower_url_prefix =
> flower_port = 5555
> default_queue = default
> celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
> {code}
> How do I run Airflow > 1.8 with Celery as a Redis broker? Is that possible?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)