You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hrbust-liu (via GitHub)" <gi...@apache.org> on 2025/06/26 07:42:02 UTC
[D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
GitHub user hrbust-liu created a discussion: In Celery mode of airflow, does each worker only execute one task at a time?
I have a question. In Celery mode of airflow, does each worker only execute one task at a time?
I want to change a global variable inside a task. Will it be read by other tasks running in parallel in dag? (I expect it will not be read. I understand it should be an independent process environment)
Additional explanation: PythonOperator configures python_callable inside
GitHub link: https://github.com/apache/airflow/discussions/52278
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "hrbust-liu (via GitHub)" <gi...@apache.org>.
GitHub user hrbust-liu added a comment to the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
Global variables of Python
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13582622
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
Depends on your celery setting. By default Celery creates separate fork for each worker - so they are separate interpreters, but you can run celery in the mode that they are sharing interpreter. Up to you. Look it up in celery docs
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13583301
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
Depends on your celery setting. By default Celery creates separate fork for each worker process - so they are separate interpreters, but you can run celery in the mode that they are sharing interpreter (`solo` mode). Up to you. Look it up in celery docs
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13583301
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
https://docs.celeryq.dev/en/stable/userguide/workers.html
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13583312
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "dominikhei (via GitHub)" <gi...@apache.org>.
GitHub user dominikhei added a comment to the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
One worker can also execute tasks concurrently, you can define this via [AIRFLOW__CELERY__WORKER_CONCURRENCY](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#worker-concurrency) and [AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-active-tasks-per-dag).
What type of variable are you referring to? Airflow variables `airflow.model.Variable` ? Or are you referring to the `global` keyword in python?
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13582372
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "hrbust-liu (via GitHub)" <gi...@apache.org>.
GitHub user hrbust-liu added a comment to the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
I understand that AIRFLOW__CELERY__WORKER_CONCURRENCY represents the number of workers that can be executed at the same time. This may indicate the number of tasks/Operators that can be run at the same time
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG represents how many tasks of a dag can run in parallel.
It seems to be different from what I want to say. What I want is that, for example, if a dag is A, B, C, D, then A/B/C/D may run at the same time. Will they be on the same process, causing conflicts when updating global variables?
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13582646
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] In Celery mode of airflow, does each worker only execute one task at a time? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk edited a comment on the discussion: In Celery mode of airflow, does each worker only execute one task at a time?
Depends on your celery setting. By default Celery creates separate fork for each worker process - so they are separate interpreters, but you can run celery in the mode that they are sharing interpreter. Up to you. Look it up in celery docs
GitHub link: https://github.com/apache/airflow/discussions/52278#discussioncomment-13583301
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org