You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zachliu (via GitHub)" <gi...@apache.org> on 2025/04/23 19:14:01 UTC
[I] DAG-processor RuntimeError: can't start new thread [airflow]
zachliu opened a new issue, #49650:
URL: https://github.com/apache/airflow/issues/49650
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
The dag-processor has `RuntimeError: can't start new thread` when the it keeps running for over 6~7 hours. I suspect there might be a memory-or-something-else-leak.
```
processor-1 | [2025-04-23T12:31:47.475+0000] {dag_processor_job_runner.py:63} ERROR - Exception when executing DagProcessorJob
processor-1 | + Exception Group Traceback (most recent call last):
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
processor-1 | | yield
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in __call__
processor-1 | | async with anyio.create_task_group() as task_group:
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
processor-1 | | raise BaseExceptionGroup(
processor-1 | | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
processor-1 | +-+---------------- 1 ----------------
processor-1 | | Traceback (most recent call last):
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1 | | await self.app(scope, receive, _send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1 | | with recv_stream, send_stream, collapse_excgroups():
processor-1 | | ^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1 | | self.gen.throw(value)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1 | | raise exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1 | | response = await self.dispatch_func(request, call_next)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1 | | response = await call_next(request)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1 | | raise app_exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1 | | await self.app(scope, receive_or_disconnect, send_no_error)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1 | | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | | raise exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | | await app(scope, receive, sender)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1 | | await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1 | | await route.handle(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1 | | await self.app(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1 | | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | | raise exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | | await app(scope, receive, sender)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1 | | response = await f(request)
processor-1 | | ^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1 | | solved_result = await solve_dependencies(
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1 | | solved = await run_in_threadpool(call, **solved_result.values)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1 | | return await anyio.to_thread.run_sync(func)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1 | | return await get_async_backend().run_sync_in_worker_thread(
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1 | | worker.start()
processor-1 | | File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1 | | _start_new_thread(self._bootstrap, ())
processor-1 | | RuntimeError: can't start new thread
processor-1 | +------------------------------------
processor-1 |
processor-1 | During handling of the above exception, another exception occurred:
processor-1 |
processor-1 | Traceback (most recent call last):
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1 | await self.app(scope, receive, _send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1 | with recv_stream, send_stream, collapse_excgroups():
processor-1 | ^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1 | self.gen.throw(value)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1 | raise exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1 | response = await self.dispatch_func(request, call_next)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1 | response = await call_next(request)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1 | raise app_exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1 | await self.app(scope, receive_or_disconnect, send_no_error)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1 | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | raise exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | await app(scope, receive, sender)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1 | await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1 | await route.handle(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1 | await self.app(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1 | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | raise exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | await app(scope, receive, sender)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1 | response = await f(request)
processor-1 | ^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1 | solved_result = await solve_dependencies(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1 | solved = await run_in_threadpool(call, **solved_result.values)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1 | return await anyio.to_thread.run_sync(func)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1 | return await get_async_backend().run_sync_in_worker_thread(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1 | worker.start()
processor-1 | File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1 | _start_new_thread(self._bootstrap, ())
processor-1 | RuntimeError: can't start new thread
processor-1 |
processor-1 | During handling of the above exception, another exception occurred:
processor-1 |
processor-1 | Traceback (most recent call last):
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
processor-1 | self.processor.run()
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 262, in run
processor-1 | return self._run_parsing_loop()
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 349, in _run_parsing_loop
processor-1 | self._service_processor_sockets(timeout=poll_time)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 391, in _service_processor_sockets
processor-1 | need_more = socket_handler(key.fileobj)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1169, in cb
processor-1 | gen.send(line)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 554, in handle_requests
processor-1 | self._handle_request(msg, log)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 289, in _handle_request
processor-1 | var = self.client.variables.get(msg.key)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 293, in get
processor-1 | resp = self.client.get(f"variables/{key}")
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1054, in get
processor-1 | return self.request(
processor-1 | ^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
processor-1 | return copy(f, *args, **kw)
processor-1 | ^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
processor-1 | do = self.iter(retry_state=retry_state)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
processor-1 | result = action(retry_state)
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
processor-1 | self._add_action_func(lambda rs: rs.outcome.result())
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
processor-1 | return self.__get_result()
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
processor-1 | raise self._exception
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
processor-1 | result = fn(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 629, in request
processor-1 | return super().request(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 827, in request
processor-1 | return self.send(request, auth=auth, follow_redirects=follow_redirects)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
processor-1 | response = self._send_handling_auth(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
processor-1 | response = self._send_handling_redirects(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
processor-1 | response = self._send_single_request(request)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1015, in _send_single_request
processor-1 | response = transport.handle_request(request)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_transports/wsgi.py", line 138, in handle_request
processor-1 | raise seen_exc_info[1]
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/applications.py", line 259, in __call__
processor-1 | await self.__call__(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
processor-1 | await super().__call__(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
processor-1 | await self.middleware_stack(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 179, in __call__
processor-1 | response = await run_in_threadpool(self.handler, request, exc)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1 | return await anyio.to_thread.run_sync(func)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1 | return await get_async_backend().run_sync_in_worker_thread(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1 | worker.start()
processor-1 | File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1 | _start_new_thread(self._bootstrap, ())
processor-1 | RuntimeError: can't start new thread
processor-1 | 2025-04-23 12:31:47 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGTERM: -15> pid=745051 signal_sent=SIGTERM
processor-1 | [2025-04-23T12:31:47.736+0000] {process_utils.py:285} INFO - Waiting up to 5 seconds for processes to exit...
processor-1 | + Exception Group Traceback (most recent call last):
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
processor-1 | | yield
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in __call__
processor-1 | | async with anyio.create_task_group() as task_group:
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
processor-1 | | raise BaseExceptionGroup(
processor-1 | | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
processor-1 | +-+---------------- 1 ----------------
processor-1 | | Traceback (most recent call last):
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1 | | await self.app(scope, receive, _send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1 | | with recv_stream, send_stream, collapse_excgroups():
processor-1 | | ^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1 | | self.gen.throw(value)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1 | | raise exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1 | | response = await self.dispatch_func(request, call_next)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1 | | response = await call_next(request)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1 | | raise app_exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1 | | await self.app(scope, receive_or_disconnect, send_no_error)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1 | | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | | raise exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | | await app(scope, receive, sender)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1 | | await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1 | | await route.handle(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1 | | await self.app(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1 | | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | | raise exc
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | | await app(scope, receive, sender)
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1 | | response = await f(request)
processor-1 | | ^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1 | | solved_result = await solve_dependencies(
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1 | | solved = await run_in_threadpool(call, **solved_result.values)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1 | | return await anyio.to_thread.run_sync(func)
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1 | | return await get_async_backend().run_sync_in_worker_thread(
processor-1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1 | | worker.start()
processor-1 | | File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1 | | _start_new_thread(self._bootstrap, ())
processor-1 | | RuntimeError: can't start new thread
processor-1 | +------------------------------------
processor-1 |
processor-1 | During handling of the above exception, another exception occurred:
processor-1 |
processor-1 | Traceback (most recent call last):
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1 | await self.app(scope, receive, _send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1 | with recv_stream, send_stream, collapse_excgroups():
processor-1 | ^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1 | self.gen.throw(value)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1 | raise exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1 | response = await self.dispatch_func(request, call_next)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1 | response = await call_next(request)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1 | raise app_exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1 | await self.app(scope, receive_or_disconnect, send_no_error)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1 | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | raise exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | await app(scope, receive, sender)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1 | await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1 | await route.handle(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1 | await self.app(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1 | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1 | raise exc
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1 | await app(scope, receive, sender)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1 | response = await f(request)
processor-1 | ^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1 | solved_result = await solve_dependencies(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1 | solved = await run_in_threadpool(call, **solved_result.values)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1 | return await anyio.to_thread.run_sync(func)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1 | return await get_async_backend().run_sync_in_worker_thread(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1 | worker.start()
processor-1 | File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1 | _start_new_thread(self._bootstrap, ())
processor-1 | RuntimeError: can't start new thread
processor-1 |
processor-1 | During handling of the above exception, another exception occurred:
processor-1 |
processor-1 | Traceback (most recent call last):
processor-1 | File "/usr/local/airflow/.local/bin/airflow", line 8, in <module>
processor-1 | sys.exit(main())
processor-1 | ^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line 55, in main
processor-1 | args.func(args)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 48, in command
processor-1 | return func(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/utils/cli.py", line 111, in wrapper
processor-1 | return f(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
processor-1 | return func(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/dag_processor_command.py", line 54, in dag_processor
processor-1 | run_command_with_daemon_option(
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
processor-1 | callback()
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/dag_processor_command.py", line 57, in <lambda>
processor-1 | callback=lambda: run_job(job=job_runner.job, execute_callable=job_runner._execute),
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 101, in wrapper
processor-1 | return func(*args, session=session, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/job.py", line 347, in run_job
processor-1 | return execute_job(job, execute_callable=execute_callable)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/job.py", line 376, in execute_job
processor-1 | ret = execute_callable()
processor-1 | ^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
processor-1 | self.processor.run()
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 262, in run
processor-1 | return self._run_parsing_loop()
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 349, in _run_parsing_loop
processor-1 | self._service_processor_sockets(timeout=poll_time)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 391, in _service_processor_sockets
processor-1 | need_more = socket_handler(key.fileobj)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1169, in cb
processor-1 | gen.send(line)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 554, in handle_requests
processor-1 | self._handle_request(msg, log)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 289, in _handle_request
processor-1 | var = self.client.variables.get(msg.key)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 293, in get
processor-1 | resp = self.client.get(f"variables/{key}")
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1054, in get
processor-1 | return self.request(
processor-1 | ^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
processor-1 | return copy(f, *args, **kw)
processor-1 | ^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
processor-1 | do = self.iter(retry_state=retry_state)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
processor-1 | result = action(retry_state)
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
processor-1 | self._add_action_func(lambda rs: rs.outcome.result())
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
processor-1 | return self.__get_result()
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
processor-1 | raise self._exception
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
processor-1 | result = fn(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 629, in request
processor-1 | return super().request(*args, **kwargs)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 827, in request
processor-1 | return self.send(request, auth=auth, follow_redirects=follow_redirects)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
processor-1 | response = self._send_handling_auth(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
processor-1 | response = self._send_handling_redirects(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
processor-1 | response = self._send_single_request(request)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1015, in _send_single_request
processor-1 | response = transport.handle_request(request)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_transports/wsgi.py", line 138, in handle_request
processor-1 | raise seen_exc_info[1]
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/applications.py", line 259, in __call__
processor-1 | await self.__call__(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
processor-1 | await super().__call__(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
processor-1 | await self.middleware_stack(scope, receive, send)
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 179, in __call__
processor-1 | response = await run_in_threadpool(self.handler, request, exc)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1 | return await anyio.to_thread.run_sync(func)
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1 | return await get_async_backend().run_sync_in_worker_thread(
processor-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1 | File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1 | worker.start()
processor-1 | File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1 | _start_new_thread(self._bootstrap, ())
processor-1 | RuntimeError: can't start new thread
```
### What you think should happen instead?
The dag-processor should be stable enough to run indefinitely
### How to reproduce
Keep running dag-processor for long period of time.
### Operating System
LM 21.3 with docker python:3.12-slim-bookworm
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-processor RuntimeError: can't start new thread [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #49650:
URL: https://github.com/apache/airflow/issues/49650#issuecomment-2834038669
Is more log available right before this happens?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-processor RuntimeError: can't start new thread [airflow]
Posted by "zachliu (via GitHub)" <gi...@apache.org>.
zachliu commented on issue #49650:
URL: https://github.com/apache/airflow/issues/49650#issuecomment-2836013393
this is the cpu/memory metrics of the dag-processor at the time when the process reboots (not the k8s pod but the process `exec airflow dag-processor` itself)
damn thing is i couldn't find any error messages at these moments. i searched all these keywords:
* oom
* errno
* health
* heart
* failed
* lost
* zombie
* exit
* etc.
almost like something scheduled the reboots but my `[dag_processor]` section of the `airflow.cfg` is entirely default: https://github.com/apache/airflow/blob/40cb51d135094c6eca7c69e75fb192f9d1113bde/airflow-core/src/airflow/config_templates/config.yml#L2426-L2597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-processor RuntimeError: can't start new thread [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #49650:
URL: https://github.com/apache/airflow/issues/49650#issuecomment-2833993764
cc @uranusjr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-processor RuntimeError: can't start new thread [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #49650:
URL: https://github.com/apache/airflow/issues/49650#issuecomment-2834029090
This happens when a process has too many active threads the OS allows no more to be created. This error is just a consequence, we need to instead look into where we spawn threads and where we end them instead. This will be a fun one to debug…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-processor RuntimeError: can't start new thread [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #49650:
URL: https://github.com/apache/airflow/issues/49650#issuecomment-2834090062
https://forums.docker.com/t/138142
This suggests it could be a Docker issue and you may need to add `--privileged` so the container have access to more thread capacity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org