You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "fabbuc-gyg (via GitHub)" <gi...@apache.org> on 2026/07/24 12:37:22 UTC

[PR] fix(databricks): sanitize failed-task error text so it can be stored in jsonb [airflow]

fabbuc-gyg opened a new pull request, #70385:
URL: https://github.com/apache/airflow/pull/70385

   ## What
   
   Strip characters that Postgres `jsonb` cannot store (NUL bytes and unpaired UTF-16 surrogates) from a failed Databricks task's error text, in both `extract_failed_task_errors` and `extract_failed_task_errors_async`.
   
   ## Why
   
   `DatabricksExecutionTrigger` puts each failed task's error (taken from `hook.get_run_output(...)["error"]`, i.e. arbitrary external text) into its `TriggerEvent`. Airflow persists that event payload into the deferred task instance's `next_kwargs`, which is a Postgres `jsonb` column.
   
   Databricks run output can contain NUL bytes or unpaired surrogates (for example when a task emits binary data — we hit this in production where a run's error contained a ZIP/`PK\x03\x04...` payload). Postgres `jsonb` rejects these, so the triggerer crashes when it writes the event:
   
   ```
   sqlalchemy.exc.DataError: (psycopg2.errors.UntranslatableCharacter) unsupported Unicode escape sequence
   DETAIL: \u0000 cannot be converted to text.
   ```
   
   Because the invalid event is retried, the triggerer enters a `CrashLoopBackOff`, which stalls **all** deferrable tasks on the deployment, not just the offending one.
   
   ## How
   
   A small `make_jsonb_safe()` helper regex-strips `\x00` and `\ud800-\udfff` from the error string (non-string values pass through unchanged). It is applied to the extracted error in both the sync and async helpers, so the emitted trigger event is always persistable.
   
   ## Tests
   
   Added unit tests in `providers/databricks/tests/unit/databricks/utils/test_databricks.py`:
   - direct `make_jsonb_safe` cases (NUL strip, unpaired-surrogate strip, clean-text passthrough, non-string passthrough)
   - `extract_failed_task_errors` and `extract_failed_task_errors_async` sanitize error output
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes - Cursor was used to help draft this change; the author reviewed it.
   
   Generated-by: Cursor
   
   Made with [Cursor](https://cursor.com)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub)" <gi...@apache.org>.
fabbuc-gyg commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5540277910

   Thanks @moomindani for the thorough verification, and sorry for the slow turnaround @eladkal.
   
   Both open items are addressed, and the branch is rebased on current `main`:
   
   - **D200 (blocker):** docstring is now a single line. Verified with the pinned `ruff==0.16.4` from `.pre-commit-config.yaml` — `ruff check` and `ruff format --check` both pass on the two touched files. `prek` is installed locally now.
   - **Annotation:** `error: str | int -> str | int` is now `error: Any -> Any`. You were right that `str | int` implied `int` was meaningful when it was only incidental; the guard is there to be defensive about whatever Databricks returns in `run_output["error"]`, and `Any` says that honestly. Behaviour is unchanged, so the existing tests (including the non-string passthrough case) still cover it.
   
   No changes to the sanitizer logic or its scoping — appreciated the confirmation that `run_state.state_message` is already safe via `RunState.to_json()` and that the character class is not over-broad.
   
   CI on the new commit is sitting in `action_required`, so it needs a committer to approve the run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5284967303

   @fabbuc-gyg can you please fix the open items?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on code in PR #70385:
URL: https://github.com/apache/airflow/pull/70385#discussion_r3670375957


##########
providers/databricks/src/airflow/providers/databricks/utils/databricks.py:
##########
@@ -17,9 +17,22 @@
 # under the License.
 from __future__ import annotations
 
+import re
+
 from airflow.providers.common.compat.sdk import AirflowException, XComArg
 from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
 
+_JSONB_INVALID_CHARS = re.compile(r"[\x00\ud800-\udfff]")
+
+
+def make_jsonb_safe(error: str | int) -> str | int:
+    """

Review Comment:
   `ruff` fails here with `D200 One-line docstring should fit on one line`, which is what the CI Static checks job is failing on. `main` passes this hook, so it comes from this PR.
   
   ```python
   def make_jsonb_safe(error: str | int) -> str | int:
       """Strip characters that cannot be stored in a Postgres ``jsonb`` column from error text."""
   ```
   
   As a first-time contributor you may not have the hooks installed yet — `uv tool install prek && prek install` will run these locally before each commit and catch this class of failure before pushing. See [static code checks](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
   



##########
providers/databricks/src/airflow/providers/databricks/utils/databricks.py:
##########
@@ -17,9 +17,22 @@
 # under the License.
 from __future__ import annotations
 
+import re
+
 from airflow.providers.common.compat.sdk import AirflowException, XComArg
 from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
 
+_JSONB_INVALID_CHARS = re.compile(r"[\x00\ud800-\udfff]")
+
+
+def make_jsonb_safe(error: str | int) -> str | int:

Review Comment:
   Minor, non-blocking: the annotation `error: str | int` is narrower than what the function actually accepts — it passes through any non-string unchanged, and callers here only ever pass `run_output["error"]` or `state_message`, both of which are strings in practice. Since the guard exists to be defensive about whatever Databricks returns, `error: Any -> Any` (or just `str` if you want to assert the contract) would describe the behaviour more honestly than `str | int`, which suggests `int` is a meaningful case when it is only incidental.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5736877705

   Nothing from my side — the approval stands, and the rebase I flagged in my merge-order note is not needed.
   
   #72313 landed since then, and the two changes sit on different lines: it rewrote the loop header (`_latest_attempts(...)`), while this one rewrites the `failed_tasks.append(...)` payload. I merged current `main` into `f1bb073570` locally to be sure — no conflict, and the combined `utils/test_databricks.py` is 24 passed, with both the attempt-dedup tests and the sanitization tests green.
   
   So it is waiting on a committer to merge.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @moomindani before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "github-actions[bot] (via GitHub) via commits" <co...@airflow.apache.org>.
github-actions[bot] commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-6591894767

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "eladkal (via GitHub) via commits" <co...@airflow.apache.org>.
eladkal merged PR #70385:
URL: https://github.com/apache/airflow/pull/70385


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(databricks): sanitize failed-task error text so it can be stored in jsonb [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5069917620

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub)" <gi...@apache.org>.
fabbuc-gyg commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5713601399

   Is there anything blocking this from a merge? :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5552028392

   The production case is useful — an independent report with a real `AnalysisException` payload is better evidence for this fix than my synthetic one.
   
   The second hole is not there, though, and I checked rather than reasoned. I replayed both payload shapes into a real Postgres 16.13 `jsonb` column the way SQLAlchemy writes them (`json.dumps` text cast to `jsonb`), with the same dirty string — a NUL plus a lone surrogate — in each:
   
   | payload | result |
   |---|---|
   | `errors[].error`, unsanitized | rejected: `UntranslatableCharacter: unsupported Unicode escape sequence` |
   | `run_state = RunState.to_json()` carrying that `state_message` | **stored** |
   | `run_state` as a nested dict instead of a `to_json()` string | rejected |
   
   The double encoding is what saves it. `json.dumps` turns a NUL into the six ASCII characters `\u0000`; serializing the event payload for the column escapes that backslash in turn, so Postgres sees literal text rather than an escape it must interpret. Rejection needs a *live* `\u0000` in the wire JSON, and only the raw `errors[]` path produces one — the third row is what that would take.
   
   I suspect the repro is what led there: in Python `"boom \^@ more"` is a literal backslash-caret-at rather than a NUL, since `\^` is not an escape sequence — `\^@` is how psql renders a NUL, not how Python writes one. A real `\x00` comes out as `\u0000`. Every state in these triggers goes through `to_json()` (`triggers/databricks.py:155`, `:181`, `:208`, `:320`, `:331`, `:424`, and `operators/databricks.py:280`), so `ClusterState` and `SQLStatementState` are covered by the same stringification.
   
   One thing does remain, and it is not this: `from_json()` revives the real NUL, so if a revived `state_message` is ever put into a payload unstringified, that is the hole. Today it only reaches exception text and logs. Sanitizing inside `to_json()` would also change the string that `databricks_repair_reason_new_settings` matching reads (`operators/databricks.py:210`, `:226`) — an argument for keeping this at the persistence boundary, where the PR already has it.
   
   So this looks right to me as it stands, and my approval holds.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @moomindani before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub)" <gi...@apache.org>.
fabbuc-gyg commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5538065982

   Hi @eladkal, So sorry for the delay. Will check this out today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "seanmuth (via GitHub) via commits" <co...@airflow.apache.org>.
seanmuth commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-6663092686

   Followup on my 9/4 review: I'm retracting the `RunState.to_json()` concern. I re-ran the escaping and @moomindani is right. `json.dumps` writes a NUL as `\u0000`, and the nested stringification escapes the backslash again, so Postgres sees literal text and `run_state` doesn't hit the jsonb error. My `\^@` repro was a literal backslash-caret-at in Python, not a NUL, so it didn't show what I thought it did. Keeping the sanitization at the persistence boundary, as this PR does, also avoids changing the string that `databricks_repair_reason_new_settings` matches on.
   
   I also re-checked the head after the 11/05 rebase. The PR's own changes (`make_jsonb_safe`, the two call sites, the tests) are identical to what was approved. The only new lines come from #72313 on `main`, and they don't overlap. I found nothing else blocking. LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub)" <gi...@apache.org>.
fabbuc-gyg commented on code in PR #70385:
URL: https://github.com/apache/airflow/pull/70385#discussion_r3933816915


##########
providers/databricks/src/airflow/providers/databricks/utils/databricks.py:
##########
@@ -17,9 +17,22 @@
 # under the License.
 from __future__ import annotations
 
+import re
+
 from airflow.providers.common.compat.sdk import AirflowException, XComArg
 from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
 
+_JSONB_INVALID_CHARS = re.compile(r"[\x00\ud800-\udfff]")
+
+
+def make_jsonb_safe(error: str | int) -> str | int:
+    """

Review Comment:
   Fixed — docstring is now a single line. Also installed `prek` locally so this class of failure gets caught before pushing. Thanks for the pointer.



##########
providers/databricks/src/airflow/providers/databricks/utils/databricks.py:
##########
@@ -17,9 +17,22 @@
 # under the License.
 from __future__ import annotations
 
+import re
+
 from airflow.providers.common.compat.sdk import AirflowException, XComArg
 from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
 
+_JSONB_INVALID_CHARS = re.compile(r"[\x00\ud800-\udfff]")
+
+
+def make_jsonb_safe(error: str | int) -> str | int:

Review Comment:
   Agreed, changed to `error: Any -> Any`. `str | int` was misleading — `int` was never a meaningful case, the guard is just there to be defensive about whatever Databricks puts in `run_output["error"]`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub)" <gi...@apache.org>.
fabbuc-gyg commented on code in PR #70385:
URL: https://github.com/apache/airflow/pull/70385#discussion_r3933935628


##########
providers/databricks/src/airflow/providers/databricks/utils/databricks.py:
##########
@@ -17,9 +17,22 @@
 # under the License.
 from __future__ import annotations
 
+import re
+
 from airflow.providers.common.compat.sdk import AirflowException, XComArg
 from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
 
+_JSONB_INVALID_CHARS = re.compile(r"[\x00\ud800-\udfff]")
+
+
+def make_jsonb_safe(error: str | int) -> str | int:
+    """

Review Comment:
   Ok thanks :-)



##########
providers/databricks/src/airflow/providers/databricks/utils/databricks.py:
##########
@@ -17,9 +17,22 @@
 # under the License.
 from __future__ import annotations
 
+import re
+
 from airflow.providers.common.compat.sdk import AirflowException, XComArg
 from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
 
+_JSONB_INVALID_CHARS = re.compile(r"[\x00\ud800-\udfff]")
+
+
+def make_jsonb_safe(error: str | int) -> str | int:

Review Comment:
   Thanks, I adjusted it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub)" <gi...@apache.org>.
fabbuc-gyg commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5540406623

   Should be fine now :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-5088912307

   cc @moomindani for Databricks team review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Sanitize error message in DatabricksExecutionTrigger [airflow]

Posted by "fabbuc-gyg (via GitHub) via commits" <co...@airflow.apache.org>.
fabbuc-gyg commented on PR #70385:
URL: https://github.com/apache/airflow/pull/70385#issuecomment-6641772716

   @eladkal , Can we merge this sometime? At my company, we will need this feature..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org