You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ahilashsasidharan (via GitHub)" <gi...@apache.org> on 2026/07/31 12:38:28 UTC
[PR] Rewrite DatabricksSQLStatementsSensor exclusivity check and move back… [airflow]
ahilashsasidharan opened a new pull request, #70831:
URL: https://github.com/apache/airflow/pull/70831
… to __init__
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
## Description
This PR moves template_field exclusivity check back to `__init__` from `execute()` while keeping the `not statement and not statement_id` sibling in `execute()` since it is a required-argument check (see #70503) for the `DatabricksSQLStatementsSensor` operators in the providers/databricks/src/airflow/providers/databricks/sensors/databricks.py.
The check is updated to use `is not None` polarity rather than a truthiness check, so this is not a straight revert of the original code.
These are pure provision checks ("was this argument passed?") which belong in `__init__` per the updated guidance in #70296, as moving them to execute() risks false positives when `render_template_as_native_obj=True` renders a provided field to None.
## Tests
- Separated tests under providers/databricks/tests/unit/databricks/sensors/test_databricks.py added as part of the previous PR #70340 to validate the new behaviour.
related: #70296 and #70503
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [X] Yes (please specify the tool below)
Generated-by: [Antigravity IDE] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3887844766
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -69,15 +69,33 @@ def test_init_statement_id(self):
assert op.warehouse_id == WAREHOUSE_ID
@pytest.mark.parametrize(
- ("kwargs", "match"),
+ ("statement", "statement_id"),
[
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
+ (STATEMENT, STATEMENT_ID),
+ (STATEMENT, ""),
],
)
- def test_statement_combination_validated_at_execute(self, kwargs, match):
- op = DatabricksSQLStatementsSensor(task_id=TASK_ID, warehouse_id=WAREHOUSE_ID, **kwargs)
- with pytest.raises(AirflowException, match=match):
+ def test_both_statements_included_validated_at_init(self, statement, statement_id):
+ with pytest.raises(ValueError, match="Cannot provide both"):
+ DatabricksSQLStatementsSensor(
+ statement=statement,
+ statement_id=statement_id,
+ task_id=TASK_ID,
+ warehouse_id=WAREHOUSE_ID,
+ )
+
+ @pytest.mark.parametrize(
+ ("statement", "statement_id"),
+ [
+ (None, None),
+ ("", None),
Review Comment:
Good question, and I think the dead-code version actually weakens the guard rather than strengthening it.
If the constructor raises, `render_template_fields` and `execute` never run — so the block asserts only "something in here raised `ValueError`". Move the check back into `execute()` later and the test still passes, which is exactly the regression you want it to catch. The stronger form is the opposite: assert on the constructor **alone**, with nothing after it. Then if the check moves out of `__init__`, the constructor stops raising and the test fails immediately.
```python
with pytest.raises(ValueError, match="Provide exactly one of statement or statement_id"):
DatabricksSQLStatementsSensor(
task_id=TASK_ID, warehouse_id=WAREHOUSE_ID,
statement=statement, statement_id=statement_id, dag=dag,
)
```
That also discriminates against the old code, which is what my note was really asking for. Verified both sides locally with `render_template_as_native_obj=True`, `statement="SELECT 1"`, `statement_id="{{ None }}"`:
* current main (checks in `execute()`, truthiness): constructs fine, renders to `statement_id=None`, and `execute()` proceeds to submit the statement — no exception at all.
* this PR: `ValueError: Cannot provide both statement and statement_id.` from the constructor.
So a plain `pytest.raises` around the constructor fails before the change and passes after it, which no current test does.
Two smaller points:
The `dag=...` with `render_template_as_native_obj=True` documents the scenario but the test does not depend on it — the constructor never renders, so it behaves the same without the flag. If you want a test where rendering is load-bearing rather than dead code, that is the *other* half, and it is worth having as a second test: pass only `statement="{{ None }}"`, construct, `render_template_fields`, then assert `execute()` raises. I measured that on this PR's head and it gives `ValueError: One of either statement or statement_id must be provided.` — the render call there is doing real work, and the pair of tests then pins each check to its own site.
Also note your `match=` uses "Provide exactly one of statement or statement_id" while the code currently says "Cannot provide both statement and statement_id." — that test will fail unless the message change from my other comment lands too. Worth doing both in the same push.
---
Drafted-by: Claude Code (Opus 5); reviewed by @moomindani before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rewrite DatabricksSQLStatementsSensor exclusivity check and move back… [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5142958179
Question: Could the AirflowException for both checks mentioned in this PR be converted to ValueError raises as part of this PR or should they remain as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3892030443
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -69,15 +69,33 @@ def test_init_statement_id(self):
assert op.warehouse_id == WAREHOUSE_ID
@pytest.mark.parametrize(
- ("kwargs", "match"),
+ ("statement", "statement_id"),
[
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
+ (STATEMENT, STATEMENT_ID),
+ (STATEMENT, ""),
],
)
- def test_statement_combination_validated_at_execute(self, kwargs, match):
- op = DatabricksSQLStatementsSensor(task_id=TASK_ID, warehouse_id=WAREHOUSE_ID, **kwargs)
- with pytest.raises(AirflowException, match=match):
+ def test_both_statements_included_validated_at_init(self, statement, statement_id):
+ with pytest.raises(ValueError, match="Cannot provide both"):
+ DatabricksSQLStatementsSensor(
+ statement=statement,
+ statement_id=statement_id,
+ task_id=TASK_ID,
+ warehouse_id=WAREHOUSE_ID,
+ )
+
+ @pytest.mark.parametrize(
+ ("statement", "statement_id"),
+ [
+ (None, None),
+ ("", None),
Review Comment:
Thanks I added both testcases. I guess I was overthinking this with the dead code idea, but I believe covering the _init_ template check should be enough and I added a comment for intent. Let me know if there are any issues with my implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5223062424
@shahar1 pinging since this should be ready for a re-review when you have the chance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5400734833
@ahilashsasidharan can you fix the open items?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3810490154
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -69,15 +69,33 @@ def test_init_statement_id(self):
assert op.warehouse_id == WAREHOUSE_ID
@pytest.mark.parametrize(
- ("kwargs", "match"),
+ ("statement", "statement_id"),
[
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
+ (STATEMENT, STATEMENT_ID),
+ (STATEMENT, ""),
Review Comment:
Worth adding `("", "")` here so whichever behaviour you settle on for the both-empty case is pinned down — right now nothing covers it.
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -69,15 +69,33 @@ def test_init_statement_id(self):
assert op.warehouse_id == WAREHOUSE_ID
@pytest.mark.parametrize(
- ("kwargs", "match"),
+ ("statement", "statement_id"),
[
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
+ (STATEMENT, STATEMENT_ID),
+ (STATEMENT, ""),
],
)
- def test_statement_combination_validated_at_execute(self, kwargs, match):
- op = DatabricksSQLStatementsSensor(task_id=TASK_ID, warehouse_id=WAREHOUSE_ID, **kwargs)
- with pytest.raises(AirflowException, match=match):
+ def test_both_statements_included_validated_at_init(self, statement, statement_id):
+ with pytest.raises(ValueError, match="Cannot provide both"):
+ DatabricksSQLStatementsSensor(
+ statement=statement,
+ statement_id=statement_id,
+ task_id=TASK_ID,
+ warehouse_id=WAREHOUSE_ID,
+ )
+
+ @pytest.mark.parametrize(
+ ("statement", "statement_id"),
+ [
+ (None, None),
+ ("", None),
Review Comment:
The case that actually motivates the polarity change is not covered anywhere: a *provided* field that renders to `None` under `render_template_as_native_obj=True`, which must not slip past the exclusivity check. Without it, a later well-meaning move back to a truthiness check in `execute()` would pass CI.
##########
providers/databricks/src/airflow/providers/databricks/sensors/databricks.py:
##########
@@ -69,9 +69,11 @@ def __init__(
include_airflow_query_tags: bool = True,
**kwargs,
):
- # Handle the scenario where either both statement and statement_id are set/not set
+ if statement is not None and statement_id is not None:
Review Comment:
With both fields `""` this raises "Cannot provide both" even though nothing was really provided — measured on the patched build, and `main` reports the accurate "One of either statement or statement_id must be provided." from `execute()` for the same input.
The `is not None` polarity is right and I would not weaken it; the cheapest fix is a message that holds for either failure, e.g. `"Provide exactly one of statement or statement_id."`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3892018582
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -69,15 +69,33 @@ def test_init_statement_id(self):
assert op.warehouse_id == WAREHOUSE_ID
@pytest.mark.parametrize(
- ("kwargs", "match"),
+ ("statement", "statement_id"),
[
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
+ (STATEMENT, STATEMENT_ID),
+ (STATEMENT, ""),
Review Comment:
Done. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3892020376
##########
providers/databricks/src/airflow/providers/databricks/sensors/databricks.py:
##########
@@ -69,9 +69,11 @@ def __init__(
include_airflow_query_tags: bool = True,
**kwargs,
):
- # Handle the scenario where either both statement and statement_id are set/not set
+ if statement is not None and statement_id is not None:
Review Comment:
Done. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3882872833
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -69,15 +69,33 @@ def test_init_statement_id(self):
assert op.warehouse_id == WAREHOUSE_ID
@pytest.mark.parametrize(
- ("kwargs", "match"),
+ ("statement", "statement_id"),
[
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
+ (STATEMENT, STATEMENT_ID),
+ (STATEMENT, ""),
],
)
- def test_statement_combination_validated_at_execute(self, kwargs, match):
- op = DatabricksSQLStatementsSensor(task_id=TASK_ID, warehouse_id=WAREHOUSE_ID, **kwargs)
- with pytest.raises(AirflowException, match=match):
+ def test_both_statements_included_validated_at_init(self, statement, statement_id):
+ with pytest.raises(ValueError, match="Cannot provide both"):
+ DatabricksSQLStatementsSensor(
+ statement=statement,
+ statement_id=statement_id,
+ task_id=TASK_ID,
+ warehouse_id=WAREHOUSE_ID,
+ )
+
+ @pytest.mark.parametrize(
+ ("statement", "statement_id"),
+ [
+ (None, None),
+ ("", None),
Review Comment:
I believe the polarity change should be covered, but I agree that the `render_template_as_native_obj=True` scenario isn't covered.
I am not sure how to cover it cleanly though currently I've settled on potentially implementing something like the below testcase if it passes all checks. From my understanding (AI-assisted) we need to explicitly render templates after setting the value of it in the DAG so it requires leaving dead code at the bottom within the pytest.raise block to render templates that will never run since currently `__init__` catches the error:
```python
context = {"dag": dag}
op.render_template_fields(context)
op.execute(context)
```
Without it someone moving the code block to execute might rewrite the test without rendering the templates which would still allow it to pass CI and might be missed by a reviewer. With this code they would need to explicitly remove the render which is easier to catch in a review.
I'll also add a comment mentioning intent of the test for anyone that is reading the code, but not sure if there is a cleaner way to implement this beyond that which you might have in mind.
```python
@pytest.mark.parametrize("statement,statement_id", [
("{{ None }}", STATEMENT_ID), # statement would render to None
(STATEMENT, "{{ None }}"), # statement_id would render to None
("{{ None }}", "{{ None }}"), # both would render to None
])
def test_both_provided_one_as_template_raises_at_init(self, statement, statement_id):
dag = DAG(
dag_id="test_native_obj_dag",
start_date=timezone.datetime(2025, 1, 1),
schedule=None,
render_template_as_native_obj=True,
)
with pytest.raises(ValueError, match="Provide exactly one of statement or statement_id"):
op = DatabricksSQLStatementsSensor(
task_id=TASK_ID,
warehouse_id=WAREHOUSE_ID,
statement=statement,
statement_id=statement_id,
dag=dag,
)
context = {"dag": dag}
op.render_template_fields(context)
op.execute(context)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rewrite DatabricksSQLStatementsSensor exclusivity check and move back… [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3694875677
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -68,16 +68,18 @@ def test_init_statement_id(self):
assert op.statement_id == STATEMENT_ID
assert op.warehouse_id == WAREHOUSE_ID
- @pytest.mark.parametrize(
- ("kwargs", "match"),
- [
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
- ],
- )
- def test_statement_combination_validated_at_execute(self, kwargs, match):
- op = DatabricksSQLStatementsSensor(task_id=TASK_ID, warehouse_id=WAREHOUSE_ID, **kwargs)
- with pytest.raises(AirflowException, match=match):
+ def test_both_statements_included_validated_at_init(self):
+ with pytest.raises(ValueError, match="Cannot provide both"):
+ DatabricksSQLStatementsSensor(
+ statement=STATEMENT,
+ statement_id=STATEMENT_ID,
+ task_id=TASK_ID,
+ warehouse_id=WAREHOUSE_ID,
+ )
Review Comment:
The `is not None` polarity is the headline of this PR, and nothing here exercises it. `STATEMENT` and `STATEMENT_ID` are both non-empty strings, so a plain `if statement and statement_id:` truthiness check passes this test identically — meaning the test does not fail without the change it is meant to cover, which is the bar in [Testing Standards](https://github.com/apache/airflow/blob/main/CLAUDE.md#testing-standards) ("every test must fail without the PR's change").
The distinguishing case is the empty string — exactly the one called out as the reason for the polarity in the first place. Adding it back as a parametrize keeps the split-by-lifecycle naming intact:
```suggestion
@pytest.mark.parametrize(
("statement", "statement_id"),
[
(STATEMENT, STATEMENT_ID),
(STATEMENT, ""),
],
)
def test_both_statements_included_validated_at_init(self, statement, statement_id):
with pytest.raises(ValueError, match="Cannot provide both"):
DatabricksSQLStatementsSensor(
statement=statement,
statement_id=statement_id,
task_id=TASK_ID,
warehouse_id=WAREHOUSE_ID,
)
```
The second case fails on `main` and passes with this PR, which is the property that was missing.
##########
providers/databricks/src/airflow/providers/databricks/sensors/databricks.py:
##########
@@ -107,8 +110,7 @@ def _get_hook(self, caller: str) -> DatabricksHook:
)
def execute(self, context: Context):
- if self.statement and self.statement_id:
- raise AirflowException("Cannot provide both statement and statement_id.")
+ # Handle the scenario where both statement and statement_id are not set
Review Comment:
Minor, non-blocking. This comment and its counterpart above the `__init__` check both restate the condition on the line beneath them, which [AGENTS.md](https://github.com/apache/airflow/blob/main/CLAUDE.md#coding-standards) asks us to avoid ("do not write narrating comments that restate the next line").
The thing that genuinely warrants a comment here is the part the code cannot say for itself: *why* one half of this validation lives in `__init__` and the other half lives in `execute()`. That asymmetry is deliberate and load-bearing, and it is precisely what has been getting reverted back and forth across this series — a reader arriving cold will otherwise read it as an oversight and "fix" it again.
```suggestion
# Both fields are templated, so "neither resolves to a value" is only knowable
# after rendering — __init__ cannot catch it. The both-provided case is a pure
# provision error and is checked there instead.
```
Dropping the `# Handle the scenario where both statement and statement_id are set` line above the `__init__` check would round this out — with the rationale recorded here, that one carries nothing the `if` does not already say.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5313259755
cc @moomindani for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #70831:
URL: https://github.com/apache/airflow/pull/70831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5162418061
Follow up question for some of my related commits in this series:
#70874 & #70634
1. Both make similar changes to different provider files but do not have testcases that fail without the PR and emphasise the specific logic for each check being written as it is (i.e. with `is not None` or without `is None`). Is it worth going back to add additional parameterised testcases for these?
2. Should these also have a comments to highlight the split between sibling validation to avoid future confusion by readers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rewrite DatabricksSQLStatementsSensor exclusivity check and move back… [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5149226791
This gets the split right, which is worth saying explicitly because several PRs in this series have not.
The "both provided" case is a provision check — passing both `statement` and `statement_id` is a Dag-authoring error whatever they render to — so `__init__` with `is not None` polarity is the correct home, and that polarity also catches `statement_id=""`, which a truthiness check would let through.
Leaving the "neither resolves" case in `execute()` is right too, and I checked rather than assumed: `template_fields` includes both `statement` and `statement_id`, so `statement="{{ params.sql }}"` rendering to an empty string is only visible after rendering. `__init__` genuinely cannot catch that one. The asymmetry is correct, not an oversight.
Tests are split to match, which reads well.
One thing before merge. The relocated line still raises `AirflowException`:
```python
if statement is not None and statement_id is not None:
raise AirflowException("Cannot provide both statement and statement_id.")
```
Moving one verbatim during a refactor is explicitly allowed, and the file's ratchet entry stays at `::4`, so nothing is violated as it stands. But the guidance is to narrow when you're already touching the line, and this is an argument-combination error, so `ValueError` fits — exactly what you did in #70634:
```python
if statement is not None and statement_id is not None:
raise ValueError("Cannot provide both statement and statement_id.")
```
That needs the test's `pytest.raises(AirflowException, ...)` updated to `ValueError`, and `generated/known_airflow_exceptions.txt` dropped from `providers/databricks/src/airflow/providers/databricks/sensors/databricks.py::4` to `::3`.
Happy to merge once that's in.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5412404766
> @ahilashsasidharan can you fix the open items?
Sorry for the delay. Will work on these today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5474452130
> Thanks for the clear write-up — the reasoning matches the criterion in #70503 exactly, and it follows the same restoration already merged for the Repos operators in #70551.
>
> I ran a differential against a live SQL warehouse. Provider 7.18.1 already contains #70340, so the unpatched wheel reproduces current `main` and the patched one carries your two edits (Airflow 3.2.2, real `warehouse_id`, real statements executed):
> case current main with this PR
> `statement` + templated `statement_id` rendering to `None` (`render_template_as_native_obj=True`) task **succeeds** — the contradictory pair is silently ignored and the statement runs Dag import error
> `statement="SELECT 1"`, `statement_id=""` task **succeeds** against the warehouse Dag import error
> `statement=""`, `statement_id=""` `execute()`: "One of either statement or statement_id must be provided." `__init__`: "Cannot provide both statement and statement_id."
> `statement` only / `statement_id` only constructed constructed
>
> The first row is an argument for this change that the description does not make: on `main`, supplying both arguments produces no error at all once one of them renders to `None`, and the sensor quietly runs the other. Catching that at parse time is a real improvement rather than a relocation — worth saying in the description.
>
> The second row is a user-visible behaviour change: `statement_id=""` alongside a `statement` works today (confirmed end to end) and becomes a Dag import error here. Your parametrization shows it is deliberate, so this is only about making it visible to whoever reads the changelog later.
>
> Nits inline, none blocking.
>
> Drafted-by: Claude Code (Opus 5); reviewed by @moomindani before posting
Thanks! I've updated the PR description to note both changes. Let me know if anything else is needed to make it visible in the changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rewrite DatabricksSQLStatementsSensor exclusivity check and move back… [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3701291900
##########
providers/databricks/tests/unit/databricks/sensors/test_databricks.py:
##########
@@ -68,16 +68,18 @@ def test_init_statement_id(self):
assert op.statement_id == STATEMENT_ID
assert op.warehouse_id == WAREHOUSE_ID
- @pytest.mark.parametrize(
- ("kwargs", "match"),
- [
- ({"statement": STATEMENT, "statement_id": STATEMENT_ID}, "Cannot provide both"),
- ({}, "One of either statement or statement_id"),
- ],
- )
- def test_statement_combination_validated_at_execute(self, kwargs, match):
- op = DatabricksSQLStatementsSensor(task_id=TASK_ID, warehouse_id=WAREHOUSE_ID, **kwargs)
- with pytest.raises(AirflowException, match=match):
+ def test_both_statements_included_validated_at_init(self):
+ with pytest.raises(ValueError, match="Cannot provide both"):
+ DatabricksSQLStatementsSensor(
+ statement=STATEMENT,
+ statement_id=STATEMENT_ID,
+ task_id=TASK_ID,
+ warehouse_id=WAREHOUSE_ID,
+ )
Review Comment:
Done. I've changed this testcase and edited the second testcase to cover the case of why the sibling case in execute() is not converted to is None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate DatabricksSQLStatementsSensor exclusivity at __init__ [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on PR #70831:
URL: https://github.com/apache/airflow/pull/70831#issuecomment-5162349114
> Please retitle the commit as well as the PR, since the commit message is what the changelog is generated from.
I’ve updated the PR title and rewritten the first commit message to the suggested form. My understanding is that this is the commit whose message is used when generating the provider changelog, so that’s the one I corrected.
> If you'd rather keep this PR tightly scoped to what was asked for, a follow-up is fine too — just say which way you're going so the ratchet line doesn't get churned twice.
I’ve opted to make the additional changes in this PR. Since the diff already touches this function, it felt cleaner to apply the consistent ValueError treatment now rather than opening a follow‑up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rewrite DatabricksSQLStatementsSensor exclusivity check and move back… [airflow]
Posted by "ahilashsasidharan (via GitHub)" <gi...@apache.org>.
ahilashsasidharan commented on code in PR #70831:
URL: https://github.com/apache/airflow/pull/70831#discussion_r3701292489
##########
providers/databricks/src/airflow/providers/databricks/sensors/databricks.py:
##########
@@ -107,8 +110,7 @@ def _get_hook(self, caller: str) -> DatabricksHook:
)
def execute(self, context: Context):
- if self.statement and self.statement_id:
- raise AirflowException("Cannot provide both statement and statement_id.")
+ # Handle the scenario where both statement and statement_id are not set
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org