You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "abhishekbhakat (via GitHub)" <gi...@apache.org> on 2024/01/18 14:12:03 UTC
[I] Inconsistent run_ids on UI vs API creating dagRuns [airflow]
abhishekbhakat opened a new issue, #36867:
URL: https://github.com/apache/airflow/issues/36867
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
As part of this [PR](https://github.com/apache/airflow/pull/32293) we have allowed to configure run_id pattern.
Upon setting : `AIRFLOW__SCHEDULER__ALLOWED_RUN_ID_PATTERN=^.+$`
We now have Airflow UI trying to create dagruns with custom run_id from "Trigger DAG w/ config" page which has hardcoded replacement for spaces in run_id.
Airflow API `POST /dags/{dag_id}/dagRuns` on the other hand, does not do the same thing. So creating dag runs with spaces is possible with API.
Due to this above scenario, we also have an inconsistency in s3 logging the s3 logs url has `+` replaced but that only happens when when reading. The s3 objects created on s3 do have spaces and the request get_logs_with_metadata:
```
[2023-12-18T11:18:34.767+0000] {app.py:1744} ERROR - Exception on /get_logs_with_metadata [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__
result = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/astronomer/providers/logging/amazon/aws/s3.py", line 381, in list_keys_in_s3_prefix
response.raise_for_status()
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: REDACTED
```
### What you think should happen instead?
I believe we should not allow Airflow API to create run_ids with spaces at all. That will introduce the uniformity.
### How to reproduce
If I set `AIRFLOW__SCHEDULER__ALLOWED_RUN_ID_PATTERN=^.+$`
Then:
- With API I can create a run_id from API like "run id with spaces".
- But if I try the same thing from Airflow UI then it becomes "run+id+with+spaces"
### Operating System
Ubuntu 22.04.3 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.16.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-common-io==1.2.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-postgres==5.10.0
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-sqlite==3.7.0
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent run_ids on UI vs API creating dagRuns [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on issue #36867:
URL: https://github.com/apache/airflow/issues/36867#issuecomment-1898597510
UI calls go to this function:
https://github.com/apache/airflow/blob/681859c7bffabce0c294060d811db2fb16851816/airflow/www/views.py#L1941-L1951
Whereas API goes to:
https://github.com/apache/airflow/blob/681859c7bffabce0c294060d811db2fb16851816/airflow/api_connexion/endpoints/dag_run_endpoint.py#L303-L328
UI has `.replace(" ", "+") ` but API doesn't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent run_ids on UI vs API creating dagRuns [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat commented on issue #36867:
URL: https://github.com/apache/airflow/issues/36867#issuecomment-1899896590
@ephraimbuddy's PR #36877 fixes this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent run_ids on UI vs API creating dagRuns [airflow]
Posted by "abhishekbhakat (via GitHub)" <gi...@apache.org>.
abhishekbhakat closed issue #36867: Inconsistent run_ids on UI vs API creating dagRuns
URL: https://github.com/apache/airflow/issues/36867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org