You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kadubhumika (via GitHub)" <gi...@apache.org> on 2026/10/06 09:03:50 UTC
[PR] Add explicit Databricks asset examples and validation warnings [airflow]
kadubhumika opened a new pull request, #74330:
URL: https://github.com/apache/airflow/pull/74330
## Summary
Follow-up to #74191 and implementation of #74196.
Adds explicit Databricks asset examples and documentation for SQL, RunNow, and SubmitRun operators.
### Changes
- Added explicit `outlets` examples.
- Documented mapped tasks and templated names.
- Added a warning for `outlets` with `wait_for_termination=False`.
- Preserved explicit outlets, including `outlets=[]`.
- Documented submission vs. confirmed remote completion.
- Added unit tests and system examples.
## Testing
- Focused Databricks unit tests: **9 passed**
- Ruff checks: **passed**
- `git diff --cached --check`: **passed**
Related: #74191
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ X] Yes (please specify the tool below)
<!--
Generated-by: [Chat Gpt] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6414607574
> @kadubhumika I don't have write access to apache/airflow, so I can't re-run workflows. From what I can see, CI on your latest commit ([fb41b81](https://github.com/apache/airflow/commit/fb41b8139131d1dcb0f1ebcba507c30f8c9292c9)) already finished green on Oct 23, so nothing needs re-triggering. The PR is waiting on an approving review from a committer.
>
> If you need a fresh run later, pushing an empty commit or rebasing on main will start CI again, and a committer can approve the runs if they get held.
Okay Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6042008216
> Thanks @kadubhumika. Re-reviewed at [5083889](https://github.com/apache/airflow/commit/5083889efd76199751f81078e86c5a769a28cf35).
>
> Option B looks right: fire and forget keeps caller outlets (including []), warns when outlets are set, and docs say the event means submission rather than a Unity table refresh. The three warn / empty / waiting tests are solid.
>
> I would request changes on one P1:
>
> 1. P1 — Use UnityTableIdentity in examples and docs. After [Notify downstream Dags when COPY INTO writes a Unity (Databricks) table #74191](https://github.com/apache/airflow/pull/74191), the supported way to build a Unity outlet is UnityTableIdentity(...).to_asset(), which lowercases host/catalog/schema/table and normalizes the host the same way the connection does. The system example and the SQL / RunNow / SubmitRun docs currently use Asset("databricks://..."). That can drift from COPY INTO and from the Delta sensor identities, and it disagrees with the guidance already on the COPY INTO page. Please switch the examples and docs to UnityTableIdentity, for example:
> from airflow.providers.databricks.assets.databricks import UnityTableIdentity
> table = UnityTableIdentity(
> host="my-workspace.cloud.databricks.com",
> catalog="main",
> schema="default",
> table="my_airflow_table",
> )
> outlets=[table.to_asset()]
> 2. P2 — Add a blank line before the Explicit assets heading in run_now.rst and submit_run.rst so Sphinx does not glue it to the deferrable paragraph.
> 3. P2 — Soften the sync/deferrable line to “task success after remote completion”, and note that authors who need table ready should use wait_for_termination=True or the Delta table sensor ([Databricks: detect Delta table version changes (#74195). #74225](https://github.com/apache/airflow/pull/74225)). Declared outlets still do not prove a new Delta commit.
>
> Optional: the same warning block is copied in three **init** methods; a tiny shared helper would be fine once you touch that file again.
>
> CI Static checks failure looks like an unrelated snowflake ruff fix on main (timezone.utc → UTC), same as other open PRs.
>
> Happy to re-review quickly once the UnityTableIdentity examples land.
@zozo123 Sure1 Thank You!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6013072742
Hi @potiuk @ashb , the implementation for #74196 is ready for review.
I’ve added the explicit Databricks asset examples and documentation for SQL, RunNow, and SubmitRun operators, along with the warning for `wait_for_termination=False`, outlet-preservation behavior, and focused tests.
The focused unit tests and Ruff checks pass locally, and the PR CI is currently running.
Could you please review it when you get a chance? If anything needs to be changed, please let me know and I’ll address it.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on code in PR #74330:
URL: https://github.com/apache/airflow/pull/74330#discussion_r4340639789
##########
providers/databricks/tests/system/databricks/example_databricks.py:
##########
@@ -155,8 +168,9 @@
sql_statement = DatabricksSQLStatementsOperator(
task_id="sql_statement",
databricks_conn_id="databricks_default",
- statement="select * from default.my_airflow_table",
+ statement="insert into default.my_airflow_table values (3, 'test 3')",
Review Comment:
Soory for delay in message @jroachgolf84 I changed this temporarily while testing the explicit asset and warning changes in the other files, and forgot to revert it afterward. I've now restored the original `SELECT` statement. Thanks for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6366227881
Thanks @jroachgolf84 for the review. I’ve pushed the changes:
Warning message: reworded in all three operators (DatabricksSubmitRunOperator, DatabricksRunNowOperator, DatabricksSQLStatementsOperator) as suggested. It now reads: “outlets are set with wait_for_termination=False. This means that an Asset event will be emitted when the Databricks run is submitted, not when the table is refreshed.” The SQL operator says “SQL statement” instead of “Databricks run”.
Docs: removed the “does not infer which tables...” sentence from sql_statements.rst, run_now.rst and submit_run.rst. The intro now just says how to declare outlets with UnityTableIdentity.
Tests: updated the expected warning text in the three test_outlets_warn_when_not_waiting tests. The earlier CI failure (TestDatabricksSQLStatementsOperator::test_outlets_warn_when_not_waiting) was my mistake: the expected string said “Databricks run” instead of “SQL statement”. That is fixed now, and the 9 outlets tests pass locally.
Please take another look when you have time. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6138626734
@zozo123 Hi sorry for the delayed response. I was quite busy with my academic exams and papers.
I’ve addressed the three requested changes, reviewed the code and documentation, and checked the Ruff linting and formatting. I also ran the relevant tests, and all 9 targeted tests passed. I’ve pushed the changes to the PR.
Could you please review the latest changes and let me know if you have any further suggestions or if anything else needs to be fixed? I’ll be happy to address any additional feedback.
Sorry again for the delay, and thank you for your patience!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6397134452
CI passed has been waiting for a while, could a maintainer please re-trigger the workflows? @zozo123 @jroachgolf84
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #74330:
URL: https://github.com/apache/airflow/pull/74330#discussion_r4309397847
##########
providers/databricks/src/airflow/providers/databricks/operators/databricks.py:
##########
@@ -1402,6 +1408,11 @@ def __init__(
self.databricks_retry_delay = databricks_retry_delay
self.databricks_retry_args = databricks_retry_args
self.wait_for_termination = wait_for_termination
+ if self.outlets and not self.wait_for_termination:
+ self.log.warning(
+ "outlets are set with wait_for_termination=False. The asset event will mean the "
Review Comment:
I would rephrase this as:
```
`outlets` are set with `wait_for_termination=False`. This means that an Asset event will be emitted when the Databricks run is submitted, not when the table was refreshed.
```
##########
providers/databricks/src/airflow/providers/databricks/operators/databricks.py:
##########
@@ -1713,6 +1724,11 @@ def __init__(
self.databricks_retry_delay = databricks_retry_delay
self.databricks_retry_args = databricks_retry_args
self.wait_for_termination = wait_for_termination
+ if self.outlets and not self.wait_for_termination:
+ self.log.warning(
Review Comment:
Reword this in a similar way to what I did above.
##########
providers/databricks/tests/system/databricks/example_databricks.py:
##########
@@ -155,8 +168,9 @@
sql_statement = DatabricksSQLStatementsOperator(
task_id="sql_statement",
databricks_conn_id="databricks_default",
- statement="select * from default.my_airflow_table",
+ statement="insert into default.my_airflow_table values (3, 'test 3')",
Review Comment:
Just curious, why this changes here?
##########
providers/databricks/docs/operators/sql_statements.rst:
##########
@@ -109,3 +109,47 @@ An example usage of the ``DatabricksSQLStatementsSensor`` is as follows:
:language: python
:start-after: [START howto_sensor_databricks_sql_statement]
:end-before: [END howto_sensor_databricks_sql_statement]
+
+Explicit assets
Review Comment:
I don't know if this makes sense. Why does this line need to be included:
```
``DatabricksSQLStatementsOperator`` does not infer which Databricks tables a SQL
statement writes to.
```
I don't think that there are other Operators that are inferring this in the Databricks provider, is there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6406405034
@kadubhumika I don't have write access to apache/airflow, so I can't re-run workflows. From what I can see, CI on your latest commit (fb41b81) already finished green on Oct 23, so nothing needs re-triggering. The PR is waiting on an approving review from a committer.
If you need a fresh run later, pushing an empty commit or rebasing on main will start CI again, and a committer can approve the runs if they get held.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add explicit Databricks asset examples and validation warnings [airflow]
Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74330:
URL: https://github.com/apache/airflow/pull/74330#issuecomment-6323865352
@jroachgolf84 i made some chnges according to your suggestions and push pr plaese take time to rewiew and do tell me if nay next any changes nneded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org