You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "roydobbe (via GitHub)" <gi...@apache.org> on 2026/04/07 07:08:00 UTC
[I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
roydobbe opened a new issue, #64828:
URL: https://github.com/apache/airflow/issues/64828
### Under which category would you file this issue?
Providers
### Apache Airflow version
2.11.1
### What happened and how to reproduce it?
## Repair run:
Whenever a repair run is triggered, the job parameters need to be passed to the api endpoint for it to register the params for the tasks again.
I tried replicating the operator logic locally, but only when i explicitly passed the job parameters to the repair endpoint, I saw the parameters for the task it was repairing resolved.
Currently the operator only updates the job parameters as an intermediate step before triggering the repair endpoint, but doesn't pass it along in the repair_json object.
## Idempotency token
This may be expected behavior but, when we retry an airflow task with an idempotency token, it results in a successful state, even if the task its retrying has failed initially.
We would expect it to trigger a repair run if repair run is true, else we would expect a failed state.
### What you think should happen instead?
_No response_
### Operating System
_No response_
### Deployment
Astronomer
### Apache Airflow Provider(s)
databricks
### Versions of Apache Airflow Providers
databricks - 7.12
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "sydongda (via GitHub)" <gi...@apache.org>.
sydongda commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4426593756
Hi @roydobbe , params missing can be reproducible. Could you help describe more detail about the repair state issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4198742291
Can you share a snip of the code that you are running?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #64828: DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes.
URL: https://github.com/apache/airflow/issues/64828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4197161322
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "sydongda (via GitHub)" <gi...@apache.org>.
sydongda commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4406308006
@eladkal , sure, will check this one soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "roydobbe (via GitHub)" <gi...@apache.org>.
roydobbe commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4427625711
Hey @sydongda, thanks for picking this up
It’s just like I described. Whenever a job fails and a repair is triggered from airflow the operator currently doesn’t pass the initial parameters again, which results in the job running the repair with the default params of the job.
Like I showed in my example, you need to pass the initial parameters again if you want the repair run to work properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "roydobbe (via GitHub)" <gi...@apache.org>.
roydobbe commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4201052960
I just created a job that always fails with some params (python wheel task). Leaving out the job params in the second part means they are not applied.
initial run:
```python
import json
import os
import requests
from azure.identity import ClientSecretCredential
JOB_ID = 12345
TENANT_ID = os.environ.get("AZURE_TENANT_ID", "")
CLIENT_ID = os.environ.get("AZURE_CLIENT_ID", "")
CLIENT_SECRET = os.environ.get("AZURE_CLIENT_SECRET", "")
DBX_HOST = os.environ.get("DBX_HOST", "")
DBX_RESOURCE = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d" # the same for every azure tenant
credential = ClientSecretCredential(
tenant_id=TENANT_ID,
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
)
job_params = {
"execution_date": "20260101",
"environment": "dev",
"log_level": "INFO",
}
def get_headers() -> dict:
token = credential.get_token(f"{DBX_RESOURCE}/.default")
return {
"Authorization": f"Bearer {token.token}",
"Content-Type": "application/json",
}
run_now_payload = {
"job_id": JOB_ID,
"job_parameters": job_params,
}
resp = requests.post(
f"{DBX_HOST}/api/2.1/jobs/run-now",
headers=get_headers(),
json=run_now_payload,
)
resp.raise_for_status()
run_now_result = resp.json()
print("run-now result:", run_now_result)
RUN_ID = run_now_result["run_id"]
print(f"Triggered run_id: {RUN_ID}")
```
repair run:
```python
repair_payload = {
"run_id": RUN_ID,
"rerun_all_failed_tasks": True,
"job_parameters": job_params # leave out and the job params will not be applied to the repaired run.
}
resp = requests.post(
f"{DBX_HOST}/api/2.1/jobs/runs/repair",
headers=get_headers(),
json=repair_payload,
)
resp.raise_for_status()
repair_result = resp.json()
print("repair-run result:", repair_result)
REPAIR_ID = repair_result["repair_id"]
print(f"Repair started. repair_id: {REPAIR_ID} (same run_id: {RUN_ID})")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksRunNowOperator: Repair run doesnt pass job parameters and a retry on a job with an idempotency token results always results in succes. [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #64828:
URL: https://github.com/apache/airflow/issues/64828#issuecomment-4394693503
@sydongda can you check this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org