You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zmz0305 (via GitHub)" <gi...@apache.org> on 2023/08/08 21:40:51 UTC
[GitHub] [airflow] zmz0305 opened a new issue, #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
zmz0305 opened a new issue, #33236:
URL: https://github.com/apache/airflow/issues/33236
### Description
Hi team,
We are currently migrating batch data pipeline jobs from Oozie to Airflow (GCP Composer, Google’s managed Airflow service).
Due to the strict operational and auditing requirements, in our Oozie world every new deployment is a versioned folder containing all the related files and dependencies in HDFS and Oozie will reference to the folder and initiate the versioned Oozie application bits. In the meanwhile, all the old versions of Oozie artifacts will remain in the versioned folder in HDFS, and old versions of Oozie apps remain in the Oozie UI for a set period of time. This ensures that all the history are present for easy tracking and auditing purposes.
To ensure a seamless transition between old/new deployments, during the new deployment, the old version of Oozie jobs will be set to some end times based on requirements, and the new version of Oozie jobs will have some start times that are right after the end times. In this way the old/new jobs proceed without discrepancies.
However, in Airflow it seems the deployment is meant to just override the existing dag files. Also, unlike Oozie, the start/end time is not part of metadata for the dag and there is no API for modifying them. Moreover, unlike Oozie all the dependencies are in its distinct folder, in Airflow there is this shared plugins. These impose a few problems:
1. The old version of dag files are overwritten during new deployments, which makes it impossible to track “which dagrun is using which version of code”.
2. In the UI there is no trace of the old version of dag either.
3. The dags can have complicated data dependencies, a delicate start/end time management is required to ensure different version of dags are stopped and kicked off at the right time. It’s impossible to do with Airflow due to the lack of API to configure them.
4. The plugins are shared. This means all the dags need to work with the specific version of plugins, and if a subset of dags require backward-incompatible plugins changes, it will be tricky for an automated deployment system. This is not an issue in Oozie because every deployment has it’s own independent set of dependencies, which is like a python virtual env.
Currently we have hacks and workarounds to resolve each of the above by:
1. Adding a version suffix to all the file names under the `dags/` folder (dag files, configuration files, etc.), so that new deployment files do not overwrite old deployment files.
2. Add the version suffix to the dag ids, so in the UI all the dags have versions clearly shown.
3. We find that updating the environment variables during run time have an effect on the running dags. Thus, we use a set of environment variables with unique names plus version suffixes to construct the start and end time for the all dags. I.e. when we do a new deployment, we update the environment variable for the end time of the old version and set the environment variable for the new version, so that the old dags are stopped and the new dags are seamlessly started.
4. We had to create a new GCP Composer environment whenever the plugins need to change. All the new deployment will need to based on the plugins version to choose which Composer instance to be deployed into.
### Use case/motivation
Though the hacks should work, we believe it would be good for Airflow to have
1. A set of dags and all of the related files (that belongs to a pipeline or any arbitrary type of logic unit) can have some sort of independent running environment. For example (just a primitive idea):
1. We can have a layer of folders to separate dags, and Airflow can load all the dags from the folder as completely isolated set of dags.
2. The folder name can contain the version so the dags are naturally versioned.
3. In the UI it will show the folders then click into the folder it will show the dags.
4. Dags from different folders can have the same name and dag id, so ideally the names/ids in source file doesn’t need to change to avoid dags overwriting each other.
5. The folder should contain its own plugins to run the dags, so that plugins for different set of dags are detached and it avoids shared dependency issues.
2. Make the start/end time a part of the dag metadata. Create REST API and CLI for updating them during run time.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33236:
URL: https://github.com/apache/airflow/issues/33236#issuecomment-1670345719
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33236:
URL: https://github.com/apache/airflow/issues/33236#issuecomment-1673972310
> Another question, looks like the ticket you referred to is only for dag versioning. Do you have any comments for the start/end time configuration issue I described? Are there any existing ways to better resolve it or is our walk around the only way to do it?
This is part of the same AIP. The current solution is that whenever you change DAG significantly enough that it should run with the new version of your logic completely, you should create a new DAG with different dag_id..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
URL: https://github.com/apache/airflow/issues/33236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33236:
URL: https://github.com/apache/airflow/issues/33236#issuecomment-1670352956
All the problems you mentioned are part of yet-to-be-completed https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - this is an improvement proposal that has been originated once, but never completed - and waits for someone who will lead it to completion - i.e. details it and design in enough level of details to be able to pass it through discussion, start voting and get approval - folllowing the regular process we have for improvement proposals and eventually complete the implementation.
If you and your time (and obviously your company that has a business case and would like to donate engineering hours to do so) would like to take on the task and complete it - feel absolutely free. This is one of the very good ways users of Airflow can become contributors and give back for the free software. Other than that - yes - I think in the community we know all the thing that are connected with lack of completed versioning. It just needs someone to do it.
Closing as duplicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zmz0305 commented on issue #33236: Dag versioning, isolated dependencies and start/end time configuration for deployment transition
Posted by "zmz0305 (via GitHub)" <gi...@apache.org>.
zmz0305 commented on issue #33236:
URL: https://github.com/apache/airflow/issues/33236#issuecomment-1671570889
> All the problems you mentioned are part of yet-to-be-completed https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - this is an improvement proposal that has been originated once, but never completed - and waits for someone who will lead it to completion - i.e. details it and design in enough level of details to be able to pass it through discussion, start voting and get approval - folllowing the regular process we have for improvement proposals - and eventually complete the implementation.
>
> If you and your time (and obviously your company that has a business case and would like to donate engineering hours to do so) would like to take on the task and complete it - feel absolutely free. This is one of the very good ways users of Airflow can become contributors and give back for the free software and community.
>
> Other than that - yes - I think in the community we know all the thing that are connected with lack of completed versioning. It just needs someone to do it.
>
> Closing as duplicate.
@potiuk Thanks for the info. We will look into the possibility of making contributions.
Another question, looks like the ticket you referred to is only for dag versioning. Do you have any comments for the start/end time configuration issue I described? Are there any existing ways to better resolve it or our walk around is the only way so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org