You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/08 08:52:54 UTC
[GitHub] [airflow] msumit opened a new pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
msumit opened a new pull request #11344:
URL: https://github.com/apache/airflow/pull/11344
Basically fixing 2 issues:
**Issue 1**: When a DAG has the wrong schedule_interval, it should be treated as a broken dag, but it would be appearing on UI, and people can trigger it as well and wait for to get process, which would eventually never going to happen. The issue is in dagbag.py where it is being added into dagbag.dags even if the schedule_interval is incorrect.
To reproduce, put some garbage interval in a test dag.
**Issue 2**: Not sure if it was introduced recently, but IIRC scheduler used to mark dags inactive if it can't find it anymore, something which was not happening. Found that in scheduler.py, the control never comes out of `_run_scheduler_loop` method and never calls `deactivate_stale_dags` method.
To reproduce, delete a dag and wait for it to automatically disappear.
_Haven't tested with serialization ON or with Scheduler HA thing yet._
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501872025
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -1741,6 +1730,16 @@ def _run_scheduler_loop(self) -> None:
if not is_unit_test:
time.sleep(self._processor_poll_interval)
+ # Verify that all files were processed, and if so, deactivate DAGs that
+ # haven't been touched by the scheduler as they likely have been
+ # deleted.
+ if self.processor_agent.all_files_processed:
Review comment:
This is fine on master, but will need updating under a world where multiple schedulers can coexist.
Would you mind leaving this change out for two reasons: 1. It's a separate issue to the pr subject 😀 2. To avoid conflicts on scheduler ha which I'm hopeful to merge today or tomorrow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501874217
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -1741,6 +1730,16 @@ def _run_scheduler_loop(self) -> None:
if not is_unit_test:
time.sleep(self._processor_poll_interval)
+ # Verify that all files were processed, and if so, deactivate DAGs that
+ # haven't been touched by the scheduler as they likely have been
+ # deleted.
+ if self.processor_agent.all_files_processed:
Review comment:
yeah sure @ashb.. will remove this code block from the PR for now, and will try to fix it next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501850751
##########
File path: airflow/models/dagbag.py
##########
@@ -352,9 +352,9 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
dag.fileloc = filepath
try:
dag.is_subdag = False
- self.bag_dag(dag=dag, root_dag=dag)
if isinstance(dag.normalized_schedule_interval, str):
croniter(dag.normalized_schedule_interval)
+ self.bag_dag(dag=dag, root_dag=dag)
Review comment:
@kaxil found that such a testcase already exists, so added a check to assert the dagbag size on top of it.
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -1741,6 +1730,16 @@ def _run_scheduler_loop(self) -> None:
if not is_unit_test:
time.sleep(self._processor_poll_interval)
+ # Verify that all files were processed, and if so, deactivate DAGs that
+ # haven't been touched by the scheduler as they likely have been
+ # deleted.
+ if self.processor_agent.all_files_processed:
Review comment:
yeah sure @ashb.. will remove this code block from the PR for now, and will try to fix it next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501609193
##########
File path: airflow/models/dagbag.py
##########
@@ -352,9 +352,9 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
dag.fileloc = filepath
try:
dag.is_subdag = False
- self.bag_dag(dag=dag, root_dag=dag)
if isinstance(dag.normalized_schedule_interval, str):
croniter(dag.normalized_schedule_interval)
+ self.bag_dag(dag=dag, root_dag=dag)
Review comment:
Can we add a simple test case for this, i.e. to show what happens when a garbage value is provided to schedule-interval it throws an error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit merged pull request #11344: Fix DagBag bug when a Dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit merged pull request #11344:
URL: https://github.com/apache/airflow/pull/11344
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501850751
##########
File path: airflow/models/dagbag.py
##########
@@ -352,9 +352,9 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
dag.fileloc = filepath
try:
dag.is_subdag = False
- self.bag_dag(dag=dag, root_dag=dag)
if isinstance(dag.normalized_schedule_interval, str):
croniter(dag.normalized_schedule_interval)
+ self.bag_dag(dag=dag, root_dag=dag)
Review comment:
@kaxil found that such a testcase already exists, so added a check to assert the dagbag size on top of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501609193
##########
File path: airflow/models/dagbag.py
##########
@@ -352,9 +352,9 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
dag.fileloc = filepath
try:
dag.is_subdag = False
- self.bag_dag(dag=dag, root_dag=dag)
if isinstance(dag.normalized_schedule_interval, str):
croniter(dag.normalized_schedule_interval)
+ self.bag_dag(dag=dag, root_dag=dag)
Review comment:
Can we add a simple test case for this, i.e. to show what happens when a garbage value is provided to schedule-interval
##########
File path: airflow/models/dagbag.py
##########
@@ -352,9 +352,9 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
dag.fileloc = filepath
try:
dag.is_subdag = False
- self.bag_dag(dag=dag, root_dag=dag)
if isinstance(dag.normalized_schedule_interval, str):
croniter(dag.normalized_schedule_interval)
+ self.bag_dag(dag=dag, root_dag=dag)
Review comment:
Can we add a simple test case for this, i.e. to show what happens when a garbage value is provided to schedule-interval it throws an error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#issuecomment-705450955
[The Build Workflow run](https://github.com/apache/airflow/actions/runs/295168674) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on pull request #11344: Fix DagBag bug when a Dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit commented on pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#issuecomment-705990913
Checked with the serialization flag on as well, and it's working fine. Not sure why the K8s builds are failing, any idea?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit commented on pull request #11344: Fix DagBag bug when a Dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit commented on pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#issuecomment-705990913
Checked with the serialization flag on as well, and it's working fine. Not sure why the K8s builds are failing, any idea?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] msumit merged pull request #11344: Fix DagBag bug when a Dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
msumit merged pull request #11344:
URL: https://github.com/apache/airflow/pull/11344
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#issuecomment-705450955
[The Build Workflow run](https://github.com/apache/airflow/actions/runs/295168674) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501609193
##########
File path: airflow/models/dagbag.py
##########
@@ -352,9 +352,9 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
dag.fileloc = filepath
try:
dag.is_subdag = False
- self.bag_dag(dag=dag, root_dag=dag)
if isinstance(dag.normalized_schedule_interval, str):
croniter(dag.normalized_schedule_interval)
+ self.bag_dag(dag=dag, root_dag=dag)
Review comment:
Can we add a simple test case for this, i.e. to show what happens when a garbage value is provided to schedule-interval
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11344: Fix scheduler bug when a dag has invalid schedule_interval
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11344:
URL: https://github.com/apache/airflow/pull/11344#discussion_r501872025
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -1741,6 +1730,16 @@ def _run_scheduler_loop(self) -> None:
if not is_unit_test:
time.sleep(self._processor_poll_interval)
+ # Verify that all files were processed, and if so, deactivate DAGs that
+ # haven't been touched by the scheduler as they likely have been
+ # deleted.
+ if self.processor_agent.all_files_processed:
Review comment:
This is fine on master, but will need updating under a world where multiple schedulers can coexist.
Would you mind leaving this change out for two reasons: 1. It's a separate issue to the pr subject 😀 2. To avoid conflicts on scheduler ha which I'm hopeful to merge today or tomorrow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org