You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/06 13:41:02 UTC
[GitHub] [airflow] potiuk opened a new pull request #22025: Add information on DAG pausing/deactivation/deletion
potiuk opened a new pull request #22025:
URL: https://github.com/apache/airflow/pull/22025
Many of our users do not understand how DAG deactivation, deletion
works - it's quite straightforward for us, who understand how
scheduler, serialization and refresh works, but we have not really
documented everywhere what the differences are between pausing,
deactivation and actual deletion of the metadata.
It has been distributed in a few places (API documentation, comments
when you issued a "delete" UI action, but I found that there is no
single place where it is described.
This PR adds documentation in "DAG" page about it - explaining
what happens in those different states and instructing the user
to follow -> remove dag file -> wait for deactivation -> delete
pattern if they want to remove a DAG.
Related: #21864
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820246980
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
Review comment:
This is a bit confusing I think?
When we trigger a manual run from the UI it also automatically change the paused DAG to ON so it can run. If it is left in Off the dag will not be scheduled. I'm not sue if that is the same for triggering via API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820248807
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
Review comment:
I am not sure if this is even "good" - I checked that his is the behaviour in current Airflow ("development") but maybe we should actually fix it :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820268292
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
+the ``DAG`` will re-appear as Scheduler will parse the folder, only historical runs information for the
+``DAG`` will be removed (except the logs - those are only removed if you delete the log files).
+
+This all means that if you want to actually delete a ``DAG`` and its all historical metadata, you need to do
+it in two steps:
+
+* delete the ``DAG`` file from the ``DAGS_FOLDER`` and wait until it becomes inactive
Review comment:
I updated it. No wonder it is confusing to new users. I think what I came up with is much more accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820247041
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
Review comment:
I think it never deleted the DAG itself?
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
+the ``DAG`` will re-appear as Scheduler will parse the folder, only historical runs information for the
+``DAG`` will be removed (except the logs - those are only removed if you delete the log files).
+
+This all means that if you want to actually delete a ``DAG`` and its all historical metadata, you need to do
+it in two steps:
+
+* delete the ``DAG`` file from the ``DAGS_FOLDER`` and wait until it becomes inactive
Review comment:
Is there indication to the user when it becomes inactive?
I think the only indication for it is a the column state in the DB which is not exposed to the users via the UI.
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
Review comment:
This is a bit confusing I think?
When we trigger a manual RUN it also automatically change the paused DAG to ON so it can run. If it is left in Off the dag will not be scheduled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820266268
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
+the ``DAG`` will re-appear as Scheduler will parse the folder, only historical runs information for the
+``DAG`` will be removed (except the logs - those are only removed if you delete the log files).
+
+This all means that if you want to actually delete a ``DAG`` and its all historical metadata, you need to do
+it in two steps:
+
+* delete the ``DAG`` file from the ``DAGS_FOLDER`` and wait until it becomes inactive
Review comment:
Ah no... let me correct that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820266226
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
Review comment:
Actually I got it wrong :).. This is really misleading for new people. I will correct it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820248643
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
+the ``DAG`` will re-appear as Scheduler will parse the folder, only historical runs information for the
+``DAG`` will be removed (except the logs - those are only removed if you delete the log files).
+
+This all means that if you want to actually delete a ``DAG`` and its all historical metadata, you need to do
+it in two steps:
+
+* delete the ``DAG`` file from the ``DAGS_FOLDER`` and wait until it becomes inactive
Review comment:
It is in the "All" tab and missing in teh "Active" tab.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#issuecomment-1060023971
I added a proposal to rename the "tab" to something else than active @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#issuecomment-1059979872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820248856
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
Review comment:
Yeah. But users are confused about it. It "DOES" look like "dag deletion" in the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#discussion_r820248856
##########
File path: docs/apache-airflow/concepts/dags.rst
##########
@@ -745,3 +745,34 @@ the dependency graph.
The dependency detector is configurable, so you can implement your own logic different than the defaults in
:class:`~airflow.serialization.serialized_objects.DependencyDetector`
+
+DAG pausing, deactivation and deletion
+--------------------------------------
+
+The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
+and finally all metadata for the ``DAG`` can be deleted.
+
+Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
+the database, but the user chose to disable it via the UI. The ``pause/unpause`` actions are available
+via UI and API. Paused ``DAG`` is not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs.
+
+Dag can be deactivated by removing them from the ``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER``
+and misses the ``DAG`` that it has seen before and stored in the database it will set is as inactive. The
+metadata and history of the ``DAG`` is kept and when the ``DAG`` is re-added to the ``DAGS_FOLDER`` it will
+be again activated. You cannot activate/deactivate ``DAG`` via UI or API, this can only be done by removing
+files from the ``DAGS_FOLDER``. In the UI, you can see inactive DAGs (in ``All`` tab) but they are not
+scheduled by the scheduler. You can also trigger them via UI for manual execution but they will not get
+executed until they ``DAG`` file that generated the dag re-appears in the ``DAGS_FOLDER`` and the ``DAG``
+becomes active. No data for historical runs of the ``DAG`` are lost when it is deactivated by the scheduler.
+
+You can also delete the ``DAG`` metadata from the metadata database using UI or API , but it does not
+necessarily deletes the ``DAG`` itself. If the ``DAG`` is in ``DAGS_FOLDER`` and you delete the metadata,
Review comment:
Yeah. But users are confused about it. It "DOES" look like "dag deletion".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22025:
URL: https://github.com/apache/airflow/pull/22025#issuecomment-1060022266
> We probably should finalize #14459 as this becomes very confusing.
It is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22025: Add information on DAG pausing/deactivation/deletion
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22025:
URL: https://github.com/apache/airflow/pull/22025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org