You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "abhishekbhakat (via GitHub)" <gi...@apache.org> on 2025/11/09 18:21:36 UTC
[PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
abhishekbhakat opened a new pull request, #58111:
URL: https://github.com/apache/airflow/pull/58111
Closes: #56529
Related: #54771
In #54771 the issues explained, also apply to the main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523304766
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1594,6 +1594,15 @@ def _mark_backfills_complete(self, session: Session = NEW_SESSION) -> None:
for b in backfills:
b.completed_at = now
+ def _safe_get_dag(self, dag_model: DagModel, purpose: str, session: Session) -> SerializedDAG | None:
+ """Safely get a DAG, logging errors if loading fails."""
+ try:
+ return _get_current_dag(dag_id=dag_model.dag_id, session=session)
Review Comment:
I'm not familiar with where else `_get_current_dag` is used, but could this change be made there instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on PR #58111:
URL: https://github.com/apache/airflow/pull/58111#issuecomment-3527756127
> This PR should be split into ~3 prs.
>
> 1. Don't crash the scheduler if the dag can't be loaded
>
> 2. The traceback vs just exception message
>
> 3. The change to do with templated start_date etc.
>
>
> (Because I think only 1 is appropriate to backport as a bug fix, the others are new features)
Agreed. But it gets very difficult to get eyes on split PRs. Unless it comes from a committer. Though if it's a hard request I'll make the PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523520548
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2107,6 +2112,9 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
elif field_name.endswith("_date"):
+ # Skip deserialization for strings containing Jinja templates
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
restricted to start_date and end_date
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523307645
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2585,7 +2593,11 @@ def _deserialize_dag_internal(
elif k == "dagrun_timeout":
v = cls._deserialize_timedelta(v)
elif k.endswith("_date"):
- v = cls._deserialize_datetime(v)
+ # Skip deserialization for strings containing Jinja templates
+ if isinstance(v, str) and cls._contains_jinja(v):
+ pass # Keep as-is for runtime rendering
Review Comment:
Ditto
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -552,7 +552,12 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
pass
except Exception as e:
self.log.exception("Failed to bag_dag: %s", dag.fileloc)
- self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
+ if self.dagbag_import_error_tracebacks:
+ self.import_errors[relative_fileloc] = traceback.format_exc(
+ limit=-self.dagbag_import_error_traceback_depth
+ )
+ else:
+ self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
Review Comment:
Sorry to be a pain, but this is likely a different change and not part of the bug fix.
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2107,6 +2112,9 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
elif field_name.endswith("_date"):
+ # Skip deserialization for strings containing Jinja templates
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
This change seems out of place -- why do we need this?
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2107,6 +2112,9 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
elif field_name.endswith("_date"):
+ # Skip deserialization for strings containing Jinja templates
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
Is this code really being hit for `op.args["end_date"]`? Rather than "_contains_jinja" the fix there should be to not hit this code path for things that aren't the top level fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat closed pull request #58111: Add scheduler guardrails to prevent deserialization errors from inval…
URL: https://github.com/apache/airflow/pull/58111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2532042941
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1599,6 +1591,19 @@ def _mark_backfills_complete(self, session: Session = NEW_SESSION) -> None:
for b in backfills:
b.completed_at = now
+ def _get_current_dag(self, dag_model: DagModel, purpose: str, session: Session) -> SerializedDAG | None:
+ """Get current DAG for a dag_model, logging errors if loading fails."""
+ try:
+ serdag = SerializedDagModel.get(dag_id=dag_model.dag_id, session=session)
+ if not serdag:
+ return None
+ serdag.load_op_links = False
+ return serdag.dag
+ except Exception as e:
+ self.log.exception(e)
+ self.log.error("Failed to load DAG '%s' for %s", dag_model.dag_id, purpose)
Review Comment:
```suggestion
self.log.exception("Failed to load DAG '%s' for %s", dag_model.dag_id, purpose)
```
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1599,6 +1591,19 @@ def _mark_backfills_complete(self, session: Session = NEW_SESSION) -> None:
for b in backfills:
b.completed_at = now
+ def _get_current_dag(self, dag_model: DagModel, purpose: str, session: Session) -> SerializedDAG | None:
+ """Get current DAG for a dag_model, logging errors if loading fails."""
+ try:
+ serdag = SerializedDagModel.get(dag_id=dag_model.dag_id, session=session)
+ if not serdag:
+ return None
+ serdag.load_op_links = False
+ return serdag.dag
+ except Exception as e:
+ self.log.exception(e)
+ self.log.error("Failed to load DAG '%s' for %s", dag_model.dag_id, purpose)
Review Comment:
Single canonical log line preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2533528481
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -552,7 +552,12 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
pass
except Exception as e:
self.log.exception("Failed to bag_dag: %s", dag.fileloc)
- self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
+ if self.dagbag_import_error_tracebacks:
+ self.import_errors[relative_fileloc] = traceback.format_exc(
+ limit=-self.dagbag_import_error_traceback_depth
+ )
+ else:
+ self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
Review Comment:
Try the dag mentioned in "How To Reproduce" sectionof this issue https://github.com/apache/airflow/issues/54771#issue-3341264288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523529945
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -552,7 +552,12 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
pass
except Exception as e:
self.log.exception("Failed to bag_dag: %s", dag.fileloc)
- self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
+ if self.dagbag_import_error_tracebacks:
+ self.import_errors[relative_fileloc] = traceback.format_exc(
+ limit=-self.dagbag_import_error_traceback_depth
+ )
+ else:
+ self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
Review Comment:
Here is a sample:
https://github.com/apache/airflow/pull/54775#issuecomment-3239011949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #58111:
URL: https://github.com/apache/airflow/pull/58111#issuecomment-3527641047
This PR should be split into ~3 prs.
1. Don't crash the scheduler if the dag can't be loaded
2. The traceback vs just exception message
3. The change to do with templated start_date etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on PR #58111:
URL: https://github.com/apache/airflow/pull/58111#issuecomment-3615626011
@ashb can you check if it looks fine now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2532043722
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2111,7 +2116,10 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
return None
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
- elif field_name.endswith("_date"):
+ elif field_name in ("start_date", "end_date"):
+ # Skip deserialization for strings containing Jinja templates only for top-level date fields
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
If the goal is to allow rendering start_date and end_date form Jinja templates, which is a new feature, this PR needs renaming. It's a much much bigger (conceptual) ask than "just don't crash".
One that I'm not sure it is worth Airflow supporting -- the scheduler is already struggling perf wise that I don't think we should blindly add this in just because a few users are trying it -- instead a better option (since this hasn't ever worked that I can see) would be to reject it at dag parsing time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58111:
URL: https://github.com/apache/airflow/pull/58111#issuecomment-3775551904
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523361911
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2107,6 +2112,9 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
elif field_name.endswith("_date"):
+ # Skip deserialization for strings containing Jinja templates
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
check `test_dag_validation_string_dates.py`
The `BarOperator` has `start_date` and `end_date` as instance attributes (which are in `template_fields`), and they contain Jinja templates like `"{{ (data_interval_start - macros.timedelta(days=11)).strftime('%Y-%m-%d') }}"`.
I can limit it to check only Top-level date fields (DAG.start_date, DAG.end_date, BaseOperator.start_date, BaseOperator.end_date) if that's preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2533582315
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2111,7 +2116,10 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
return None
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
- elif field_name.endswith("_date"):
+ elif field_name in ("start_date", "end_date"):
+ # Skip deserialization for strings containing Jinja templates only for top-level date fields
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
Yes, a try there is fine/good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2533516584
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1599,6 +1591,19 @@ def _mark_backfills_complete(self, session: Session = NEW_SESSION) -> None:
for b in backfills:
b.completed_at = now
+ def _get_current_dag(self, dag_model: DagModel, purpose: str, session: Session) -> SerializedDAG | None:
+ """Get current DAG for a dag_model, logging errors if loading fails."""
+ try:
+ serdag = SerializedDagModel.get(dag_id=dag_model.dag_id, session=session)
+ if not serdag:
+ return None
+ serdag.load_op_links = False
+ return serdag.dag
+ except Exception as e:
+ self.log.exception(e)
+ self.log.error("Failed to load DAG '%s' for %s", dag_model.dag_id, purpose)
Review Comment:
Then raising the broad exception won't be help identify why it failed.
Ref: https://github.com/apache/airflow/pull/54775#discussion_r2315308480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2533546268
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2111,7 +2116,10 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
return None
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
- elif field_name.endswith("_date"):
+ elif field_name in ("start_date", "end_date"):
+ # Skip deserialization for strings containing Jinja templates only for top-level date fields
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
Ok cool. I'll refactor the PR to just raise broken dag. But I'll recommend keep the try block on scheduler_job_runner as insurance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2532037491
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -552,7 +552,12 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
pass
except Exception as e:
self.log.exception("Failed to bag_dag: %s", dag.fileloc)
- self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
+ if self.dagbag_import_error_tracebacks:
+ self.import_errors[relative_fileloc] = traceback.format_exc(
+ limit=-self.dagbag_import_error_traceback_depth
+ )
+ else:
+ self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
Review Comment:
Can you give me a full dag I can test myself please @abhishekbhakat? (Sorry to be a pain, but my spidey sense is tingling that something else is going on here, and this is just papering over the cracks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2532043722
##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -2111,7 +2116,10 @@ def _deserialize_field_value(cls, field_name: str, value: Any) -> Any:
return None
elif field_name == "resources":
return Resources.from_dict(value) if value is not None else None
- elif field_name.endswith("_date"):
+ elif field_name in ("start_date", "end_date"):
+ # Skip deserialization for strings containing Jinja templates only for top-level date fields
+ if isinstance(value, str) and cls._contains_jinja(value):
+ return value # Keep as-is for runtime rendering
Review Comment:
If the goal is to allow rendering start_date and end_date form Jinja templates, which is a new feature, this PR needs renaming. It's a much much bigger (conceptual) ask than "just don't crash".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523396796
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1594,6 +1594,15 @@ def _mark_backfills_complete(self, session: Session = NEW_SESSION) -> None:
for b in backfills:
b.completed_at = now
+ def _safe_get_dag(self, dag_model: DagModel, purpose: str, session: Session) -> SerializedDAG | None:
+ """Safely get a DAG, logging errors if loading fails."""
+ try:
+ return _get_current_dag(dag_id=dag_model.dag_id, session=session)
Review Comment:
modified `_get_current_dag`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add scheduler guardrails to prevent deserialization errors from inval… [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on code in PR #58111:
URL: https://github.com/apache/airflow/pull/58111#discussion_r2523337456
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -552,7 +552,12 @@ def _process_modules(self, filepath, mods, file_last_changed_on_disk):
pass
except Exception as e:
self.log.exception("Failed to bag_dag: %s", dag.fileloc)
- self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
+ if self.dagbag_import_error_tracebacks:
+ self.import_errors[relative_fileloc] = traceback.format_exc(
+ limit=-self.dagbag_import_error_traceback_depth
+ )
+ else:
+ self.import_errors[relative_fileloc] = f"{type(e).__name__}: {e}"
Review Comment:
This helps raise the broken dag error on the UI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org