You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/01 10:01:26 UTC
[GitHub] [airflow] yevsh opened a new issue #18665: Deleting and copying DAG file to dags directory not triggering the DAG
yevsh opened a new issue #18665:
URL: https://github.com/apache/airflow/issues/18665
### Apache Airflow version
2.1.2
### Operating System
N/A runs as pod
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Openshift
### What happened
airflow-dags folder of airflow-scheduler pod/container mapped to external volume.
DAG file has property
'is_paused_upon_creation': False
Once I copy this DAG into the mapped directory, airflow loads it (I see it in airflow webui)and runs it.
Then it continues to run according to the schedule.
So far so good.
Now:
If I delete the DAG file from the directory, and copy again (same DAG file) airflow webui loads it but only after few minutes (keeps the execution history of previous runs),
but doesn’t start it immediately.
### What you expected to happen
Once new/old/same DAG copied to dags folder, it should appear in webui at once and tarted (if it has 'is_paused_upon_creation': False ).
With clean history - this one not mandatory.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18665: Deleting and copying DAG file from/to dags directory not triggering the DAG
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18665:
URL: https://github.com/apache/airflow/issues/18665#issuecomment-932981856
This is not how Airflow works. The DAG that disappeared and appeared again wiht the same DAG is "the same" DAG. If you want to change the structure of DAG heavily you are supposed to create a new DAG with new DAG_Id and it will have new history.
See the comment here: https://airflow.apache.org/docs/apache-airflow/stable/index.html?highlight=slowly#beyond-the-horizon
We have a DRAFT "Airlfow Improvement Proposal" to handle DAG versioning https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - but it needs a lot of work and discussion on how DAG versioning should look like in various cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #18665: Deleting and copying DAG file from/to dags directory not triggering the DAG
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #18665:
URL: https://github.com/apache/airflow/issues/18665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yevsh commented on issue #18665: Deleting and copying DAG file from/to dags directory not triggering the DAG
Posted by GitBox <gi...@apache.org>.
yevsh commented on issue #18665:
URL: https://github.com/apache/airflow/issues/18665#issuecomment-932868243
Thanks, I am aware of alternatives to trigger the DAG, I hoped their is also an OOB option that I wanted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18665: Deleting and copying DAG file from/to dags directory not triggering the DAG
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18665:
URL: https://github.com/apache/airflow/issues/18665#issuecomment-932981856
This is not how Airflow works. The DAG that disappeared and appeared again wiht the same DAG is "the same" DAG. If you want to change the structure of DAG heavily you are supposed to create a new DAG with new DAG_Id and it will have new history.
See the comment here: https://airflow.apache.org/docs/apache-airflow/stable/index.html?highlight=slowly#beyond-the-horizon
> Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.
We have a DRAFT "Airlfow Improvement Proposal" to handle DAG versioning https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - but it needs a lot of work and discussion on how DAG versioning should look like in various cases.
In fact even then when we implement it, your case will not work as you described it most likely. Airflow has no notion of files being deleted and recreated, if the DAG will appear in the same file ID it will likely never be considered as "new DAG" (this is what you effectively want). It is just the same DAG that disaappeared and appeared again. This feature is absolutely necessary to provide stability of Airflow behaviour, independently of the FileSystem behaviour. The case you describe is indistinguishable from temporary failure of the FileSystem where files will not be "visible" for some time and Airflow has to be resilient to those kind of issues.
There is another AIP https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher which is closer to what you would like to achieve, but still you would have to create a new DAG this way - the old dag with history will remain as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18665: Deleting and copying DAG file from/to dags directory not triggering the DAG
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18665:
URL: https://github.com/apache/airflow/issues/18665#issuecomment-932981856
This is not how Airflow works. The DAG that disappeared and appeared again wiht the same DAG is "the same" DAG. If you want to change the structure of DAG heavily you are supposed to create a new DAG with new DAG_Id and it will have new history.
See the comment here: https://airflow.apache.org/docs/apache-airflow/stable/index.html?highlight=slowly#beyond-the-horizon
> Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.
We have a DRAFT "Airlfow Improvement Proposal" to handle DAG versioning https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - but it needs a lot of work and discussion on how DAG versioning should look like in various cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] brki commented on issue #18665: Deleting and copying DAG file from/to dags directory not triggering the DAG
Posted by GitBox <gi...@apache.org>.
brki commented on issue #18665:
URL: https://github.com/apache/airflow/issues/18665#issuecomment-932335695
@yesvh personally I would not expect that behaviour. In fact it would be rather annoying in some situations.
Imagine that your DAGs are mounted on a network file share, and the file share is unavailable for a while due to a network issue. Your DAGs are scheduled to run once a day. They've already run today. You wouldn't want all of your DAGs to start running again when the network share is suddenly available again.
Would triggering the DAG via the CLI work for your use case?
Your using Kubernetes, so perhaps better would be triggering the DAG [via an API call]( https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org