You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/02 18:00:24 UTC
[GitHub] [airflow] KCIV opened a new issue #21276: Airflow Remote Connection Closes after 5 minutes when calling to long running API
KCIV opened a new issue #21276:
URL: https://github.com/apache/airflow/issues/21276
### Apache Airflow version
2.2.3 (latest released)
### What happened
Using request.get inside an airflow task in which the .get can take more than 5 minutes will cause a remote connection closure to happen which doesn't seem to be configurable in any sense. Even when setting .get(_url, timeout=None) causes the same remote connection closure to occur.
I have tested the endpoint using chrome, firefox, python on linux (VM), python on windows, and swagger in which none get the same remote connection failure.
### What you expected to happen
Because this connection closure only happens when using request.get inside airflow and not in any other instance, I would expect airflow to have a configuration for this timeout amount or a setting to remove a timeout (although that may be bad practice).
Airflow logs point towards the request library however it seems to be an issue in how airflow uses the library rather than the library itself.
### How to reproduce
I have the following API endpoint set up using basic fastAPI
```
@app.get('/sleeptest')
def sleep_test():
time.sleep(600)
return "completed"
```
this endpoint can be called by browser, swagger, python (linux VM) and python (Windows 10) using simple request.get(_url) and will return successfully "completed" after the 600 seconds.
However when a task in airflow is built like this
```
@task()
def get_sleep_test(resulstofprior):
returns = requests.get(api + "/sleeptest")
print(returns)
```
airflow will throw the following error after exactly 5 minutes every time.
```
Reading local file: /opt/airflow/logs/package/get_sleep_test/2022-02-01T21:21:02.173122+00:00/7.log
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: package.get_sleep_test manual__2022-02-01T21:21:02.173122+00:00 [queued]>
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1032} INFO - Dependencies all met for <TaskInstance: package.get_sleep_test manual__2022-02-01T21:21:02.173122+00:00 [queued]>
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1238} INFO -
--------------------------------------------------------------------------------
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1239} INFO - Starting attempt 7 of 7
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1240} INFO -
--------------------------------------------------------------------------------
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1259} INFO - Executing <Task(_PythonDecoratedOperator): get_sleep_test> on 2022-02-01 21:21:02.173122+00:00
[2022-02-01, 22:36:11 UTC] {standard_task_runner.py:52} INFO - Started process 2682 to run task
[2022-02-01, 22:36:11 UTC] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'package', 'get_sleep_test', 'manual__2022-02-01T21:21:02.173122+00:00', '--job-id', '58', '--raw', '--subdir', 'DAGS_FOLDER/package.py', '--cfg-path', '/tmp/tmpc70kpyas', '--error-file', '/tmp/tmp7dab7rsa']
[2022-02-01, 22:36:11 UTC] {standard_task_runner.py:77} INFO - Job 58: Subtask get_sleep_test
[2022-02-01, 22:36:11 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: package.get_sleep_test manual__2022-02-01T21:21:02.173122+00:00 [running]> on host 607ca202a787
[2022-02-01, 22:36:11 UTC] {taskinstance.py:1426} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=package
AIRFLOW_CTX_TASK_ID=get_sleep_test
AIRFLOW_CTX_EXECUTION_DATE=2022-02-01T21:21:02.173122+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-02-01T21:21:02.173122+00:00
[2022-02-01, 22:36:11 UTC] {logging_mixin.py:109} INFO - starting sleep test
[2022-02-01, 22:36:11 UTC] {logging_mixin.py:109} INFO - starting sleep test
[2022-02-01, 22:41:35 UTC] {taskinstance.py:1700} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 288, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 288, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1506, in _execute_task
result = execute_callable(context=context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/decorators/base.py", line 134, in execute
return_value = super().execute(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 174, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 185, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/package.py", line 108, in get_sleep_test
returns = requests.get(api + "/sleeptest", timeout=None)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
[2022-02-01, 22:41:35 UTC] {taskinstance.py:1277} INFO - Marking task as FAILED. dag_id=package, task_id=get_sleep_test, execution_date=20220201T212102, start_date=20220201T223611, end_date=20220201T224135
[2022-02-01, 22:41:35 UTC] {standard_task_runner.py:92} ERROR - Failed to execute job 58 for task get_sleep_test
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 288, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 288, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
args.func(args, dag=self.dag)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
error_file=args.error_file,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1506, in _execute_task
result = execute_callable(context=context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/decorators/base.py", line 134, in execute
return_value = super().execute(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 174, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 185, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/package.py", line 108, in get_sleep_test
returns = requests.get(api + "/sleeptest", timeout=None)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
[2022-02-01, 22:41:35 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2022-02-01, 22:41:35 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
I have also tried configuring the endpoint to by async with await along with asyncio.sleep() but neither fixed the issue of airflow timing out.
I have tried endless amounts of timeout changes both in airflow .env settings as well as fastAPI settings, ping, poke, retry with no change in the failure point.
### Operating System
Windows 10
### Versions of Apache Airflow Providers
Using and running dockerdesktop calling the compose using image 2.2.3
### Deployment
Docker-Compose
### Deployment details
docker engine v20.10.8
the process is docker-compose up airflow-init from the airflow project directory.
then docker-compose up to run the airflow.
### Anything else
Hopefully I have not made any grave mistakes in setting up this example, it feels extremely bare bones and easy to reproduce.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21276: Airflow Remote Connection Closes after 5 minutes when calling to long running API
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21276:
URL: https://github.com/apache/airflow/issues/21276#issuecomment-1028206997
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21276: Airflow Remote Connection Closes after 5 minutes when calling to long running API
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21276:
URL: https://github.com/apache/airflow/issues/21276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21276: Airflow Remote Connection Closes after 5 minutes when calling to long running API
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21276:
URL: https://github.com/apache/airflow/issues/21276#issuecomment-1028253199
This is likely timeout of docker desktop not Airflow. See https://stackoverflow.com/questions/36488209/how-to-override-the-default-value-of-compose-http-timeout-with-docker-compose-co
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org