[D] Add a "`DockerExecutor`" inside `CeleryExecutor` [airflow]

["hterik (via GitHub)" <gi...@apache.org>]

GitHub user hterik created a discussion: Add a "`DockerExecutor`" inside `CeleryExecutor`

### Description

When writing data pipelines, the dag code very often has a high dependency to python libraries and other tools installed in the environment. Today, when running Airflow locally with the `CeleryExecutor`, it continuously fetches and reparses dag code, but it will not reimport any other libraries that the dag depends on. The dag code basically runs in whatever environment the executor was first launched in. 

It would be very helpful if Airflow could automatically fetch a Docker image and run each taskinstance inside a new instance of it. This is how it works with the `KuberenetesExecutor` today, where it greatly simplifies deployment of updated libraries and providers. 

I'm aware of the `@task.docker`, while it sortof works, it has many rough edges that make it less of a seamless experience compared to how the `KubernetesExecutor` works.
* Imports made outside of the `@task` are not present in the task. Basically the whole task is a different process. 
* Dagrun Context and other things like `xcom_pull` is not accessible
* Common volumes, environment and other `docker run` arguments need to be set on every task, instead of once per each Executor instance. Likewise for other essentials like `auto_remove` that should not be task-unique.
* I have bad experience of the jinja template corrupting the temporary python file but can't remember exact details of this.
* Error/Exception handling is not as smooth. For example it's not possible to categorize retriable vs expected  exceptions (`AirflowFailException` is missing and `AirflowSkipException` need to be re-modeled as a exit-code)
* Logging becomes bound to the task and not the executor.

Some of the hurdles can be mitigated with a custom wrapper around `@task.docker` but it's still not as smooth as it could be. Long story short, it feels like you are working against the system instead of being supported by it. Otherwise i think the idea behind `@task.docker` is good, if you need multiple different execution environments per task, it's just that the abstraction hides a bit too much and the integration with the dagrun is missing a few pieces.

We've given up on `@task.docker` and to workaround the use case of continuously updating libraries,  we've made a background process that regularly updates and restarts the entire Docker image that runs the Airflow Celery Executor, just to get the latest environment, and then tasks are run as basic `@task`. This is far from ideal as it requires every task to complete before being able to update. It also makes QA-deployment of a pipeline much more difficult, if you want to dryrun a pipeline with its dependencies before merging it. 

(Same question also applies to the `EdgeExecutor`)

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/58687

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Add a "`DockerExecutor`" inside `CeleryExecutor` [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: Add a "`DockerExecutor`" inside `CeleryExecutor`

I think - if you have a good idea how to do it, proposing an AIP is a good start anyway.

GitHub link: https://github.com/apache/airflow/discussions/58687#discussioncomment-15098369

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Add a "`DockerExecutor`" inside `CeleryExecutor` [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: Add a "`DockerExecutor`" inside `CeleryExecutor`

Why not K8S executor? You generally want to reproduce pretty much everything K8S executor does, so it seems the easiest way for you will be to have K8S cluster and let Airflow use it to execute tasks via K8S Executor. Why do you want someone to basiclly reproduce what K8S does - i.e. run Pods scheduled in separate containers. If you want to implement any flexibility in such an executor, you will soon start reimplementing K8S.

Why? Why not K8S (and the answer - because we do not want to use K8s  is not good - essentially your feature request is to re-implement a lot of what it is).


GitHub link: https://github.com/apache/airflow/discussions/58687#discussioncomment-15078763

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Add a "`DockerExecutor`" inside `CeleryExecutor` [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: Add a "`DockerExecutor`" inside `CeleryExecutor`

BTW. I switch it to discussion. This is no-where near issue/feature. If someone would eve want to implement anything like that (though I find it hardly justifiable, because you can use K8S) - this one will have to be done through Airflow Improvement Proposal process.

GitHub link: https://github.com/apache/airflow/discussions/58687#discussioncomment-15078764

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Add a "`DockerExecutor`" inside `CeleryExecutor` [airflow]

["hterik (via GitHub)" <gi...@apache.org>]

GitHub user hterik added a comment to the discussion: Add a "`DockerExecutor`" inside `CeleryExecutor`

The `KubernetesExecutor` is great if you already have Kubernetes. On smaller on-prem deployments where you don't have AKS/GKE/EKS to run it for you, it's a huge task to run a cluster. For more traditionally managed servers (Proxmox, Ansible, Chef, Systemd, etc), the `CeleryExecutor` can be a more pragmatic starting point. 

Hence also why I'm curious how this is intended to be solved with the `EdgeExecutor`, it also exists for situations where you don't have K8s. I think it would benefit greatly by having a more integrated self-update or dryrun-with-new-libraries mechanism.

Compared with other systems, they tend to have a some story around these use-cases, for example Dagster that has `celery_docker_executor`.

GitHub link: https://github.com/apache/airflow/discussions/58687#discussioncomment-15083525

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org