You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "SzymonZy (via GitHub)" <gi...@apache.org> on 2025/05/17 20:37:59 UTC
[I] Scheduler always connect to api-server via http and port 8080 [airflow]
SzymonZy opened a new issue, #50739:
URL: https://github.com/apache/airflow/issues/50739
### Apache Airflow version
3.0.1
### If "Other Airflow 2 version" selected, which one?
3.0.1
### What happened?
Airflow installed in venv via PyPi (along official guide). When airflow-apiserver started on port other than 8080 and/or HTTPS will not be able to run any tasks. Airflow scheduler is accepting tasks from api-server but than pings back to api server always on http and port 8080 on localhost. I don't see any documentation that will allow to change this behaviour. It looks like this connection is hardcoded.
I'm using local executor.
### What you think should happen instead?
Scheduler should sent request to local executor running on api-server within port and protocol on which api server was run along with local executor.
### How to reproduce
Run api-server on port different than 8080 or with https. Run airflow scheduler. Endable debug. Try to execute any example DAG.
Look into scheduler debug log, it will put DAG into qeued status, try 4 times and fail with connection timeout/refeused error. Depending on port and protocol setup.
Documentation does not include any details how to force airflow scheduler to use proper port/protocol. I suspect it should be read out from database where airflow api-server should register itself when started.
### Operating System
SLES 15.5 and SLES 15.6
### Versions of Apache Airflow Providers
not relevant
### Deployment
Virtualenv installation
### Deployment details
Python 3.11
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "SzymonZy (via GitHub)" <gi...@apache.org>.
SzymonZy commented on issue #50739:
URL: https://github.com/apache/airflow/issues/50739#issuecomment-2889835543
It gives some progress below is diff from ariflow.cfg changes done:
`< execution_api_server_url = https://FQDN.com:7443
<
---
> # execution_api_server_url =
>
1186c1186
< base_url = https://FQDN.com:7443
---
> # base_url =
1208c1208
< port = 7443
---
> port = 8080
1233c1233
< ssl_cert = /opt/airflow/certificates/FQDN.cer
---
> ssl_cert =
1240c1240
< ssl_key = /opt/airflow/certificates/FQDN.pkcs8
---
> ssl_key =
`
and in venv:
export SSL_CERT_FILE=/opt/airflow/.venv/chain.crt
After restart of api-server and scheduler I'm getting error: Method not allowed when dag should run.
Scheduler log:
`2025-05-19 08:44:30 [info ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
[2025-05-19T08:44:30.089+0200] {_trace.py:47} DEBUG - connect_tcp.started host='FQDN.com' port=7443 local_address=None timeout=5.0 socket_options=None
[2025-05-19T08:44:30.090+0200] {_trace.py:47} DEBUG - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7ff6ba60c7d0>
[2025-05-19T08:44:30.091+0200] {_trace.py:47} DEBUG - start_tls.started ssl_context=<ssl.SSLContext object at 0x7ff6ba6a1d90> server_hostname='FQDN.com' timeout=5.0
[2025-05-19T08:44:30.093+0200] {_trace.py:47} DEBUG - start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7ff6ba4867d0>
[2025-05-19T08:44:30.094+0200] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.096+0200] {_trace.py:47} DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 405, b'Method Not Allowed', [(b'date', b'Mon, 19 May 2025 06:44:29 GMT'), (b'server', b'uvicorn'), (b'allow', b'GET'), (b'content-type', b'application/json'), (b'vary', b'Accept-Encoding'), (b'content-encoding', b'gzip'), (b'transfer-encoding', b'chunked')])
[2025-05-19T08:44:30.098+0200] {_client.py:1026} INFO - HTTP Request: PATCH https://FQDN.com:7443/task-instances/0196e744-93cc-7275-9aed-a540b238e934/run "HTTP/1.1 405 Method Not Allowed"
[2025-05-19T08:44:30.098+0200] {_trace.py:47} DEBUG - receive_response_body.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.098+0200] {_trace.py:47} DEBUG - receive_response_body.complete
[2025-05-19T08:44:30.099+0200] {_trace.py:47} DEBUG - response_closed.started
[2025-05-19T08:44:30.099+0200] {_trace.py:47} DEBUG - response_closed.complete
2025-05-19 08:44:30 [warning ] Server error [airflow.sdk.api.client] detail=None
2025-05-19 08:44:30 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=58090 signal_sent=SIGKILL
[2025-05-19T08:44:30.105+0200] {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
_execute_work(log, workload)
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/executors/local_executor.py", line 120, in _execute_work
supervise(
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 1338, in supervise
process = ActivitySubprocess.start(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 791, in start
proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 802, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 146, in start
resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1219, in patch
return self.request(
^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 338, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 680, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 827, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
raise exc
File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
hook(response)
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 117, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 107, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Method Not Allowed
Correlation-id=0196e749-6908-768c-bccf-208725217196
`
api-server (webserver) log:
`INFO: 172.28.62.131:52665 - "GET /api/v2/dags/test/dagRuns/manual__2025-05-19T06%3A39%3A13.341480%2B00%3A00/taskInstances?limit=50&offset=0&order_by=-start_date HTTP/1.1" 200 OK
INFO: 10.252.212.37:58130 - "PATCH /task-instances/0196e744-93cc-7275-9aed-a540b238e934/run HTTP/1.1" 405 Method Not Allowed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "SzymonZy (via GitHub)" <gi...@apache.org>.
SzymonZy closed issue #50739: Scheduler always connect to api-server via http and port 8080
URL: https://github.com/apache/airflow/issues/50739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #50739:
URL: https://github.com/apache/airflow/issues/50739#issuecomment-2888574739
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #50739:
URL: https://github.com/apache/airflow/issues/50739#issuecomment-2889111483
Does changing execution_api_server_url value help here?
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execution-api-server-url
> The url of the execution api server. Default is {BASE_URL}/execution/ where {BASE_URL} is the base url of the API Server. If {BASE_URL} is not set, it will use http://localhost:8080 as the default base url.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "Patchus (via GitHub)" <gi...@apache.org>.
Patchus commented on issue #50739:
URL: https://github.com/apache/airflow/issues/50739#issuecomment-2895109022
i am interested in this as well. I thought maybe these would do it , by adding in PATCH to that allow methods; but no dice. I can get everything working if i turn the webserver's SSL off by commenting out the Key / Cert; but when I add those in, I have the same issue.
`[api]
access_control_allow_headers = origin, content-type, accept
access_control_allow_methods = POST, GET, OPTIONS, DELETE
access_control_allow_origins = https://exampleclientapp1.com https://exampleclientapp2.com`
I added in the execution_api_server_url to be the https:///mydomain.com:8080 , and i added it to the origins, but still had issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "SzymonZy (via GitHub)" <gi...@apache.org>.
SzymonZy commented on issue #50739:
URL: https://github.com/apache/airflow/issues/50739#issuecomment-2895335555
At the end it turned out it was misconfiguration on my end. There is no need to set execution_api_server_url.
I will update and close tomorrow with resolution. Generally 3.0.1 documentation need imporvement with examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler always connect to api-server via http and port 8080 [airflow]
Posted by "SzymonZy (via GitHub)" <gi...@apache.org>.
SzymonZy commented on issue #50739:
URL: https://github.com/apache/airflow/issues/50739#issuecomment-2896668439
Things that are mandatory if you switch to different port and https seems to be:
base_url = https://FWDN.com:8443
port = 8443
ssl_cert = /cert_path
ssl_key = /key_path
Nothing more. execution_api_server_url should not be touched.
Incorrect or not modified base_url and port is causing method not allowed errors. Also messing with execution_api_server_url parameter bring unpredicted behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org