You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/28 04:43:58 UTC
[GitHub] [airflow] kn6405 opened a new issue #19275: Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'
kn6405 opened a new issue #19275:
URL: https://github.com/apache/airflow/issues/19275
### Apache Airflow version
2.2.0 (latest released)
### Operating System
Docker Image
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
1.2.0 helm chart and 2.2.0 Airflow
### What happened
I see scheduler pods and new triggerer pods have restarts. Just upgraded from 2.1.4 to 2.2.0.
In the triggerer pod logs:
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-10-28 04:31:26,124] {triggerer_job.py:101} INFO - Starting the triggerer
[2021-10-28 04:36:16,431] {triggerer_job.py:111} INFO - Waiting for triggers to clean up
[2021-10-28 04:36:16,431] {triggerer_job.py:117} INFO - Exited trigger loop
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/triggerer_job.py", line 231, in run
asyncio.run(self.arun()) # pylint: disable=no-member
AttributeError: module 'asyncio' has no attribute 'run'
Scheduler logs:
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-10-28 04:31:31,031] {scheduler_job.py:555} INFO - Starting the scheduler
[2021-10-28 04:31:31,031] {scheduler_job.py:560} INFO - Processing each file at most -1 times
[2021-10-28 04:31:31,258] {manager.py:163} INFO - Launched DagFileProcessorManager with pid: 31
[2021-10-28 04:31:31,264] {scheduler_job.py:1066} INFO - Resetting orphaned tasks for active dag runs
[2021-10-28 04:31:31,272] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
[2021-10-28 04:31:31,276] {settings.py:462} INFO - Loaded airflow_local_settings from /opt/airflow/config/airflow_local_settings.py .
Process DagFileProcessor322-Process:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 168, in _run_file_processor
callback_requests=callback_requests,
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 629, in process_file
self._deactivate_missing_dags(session, dagbag, file_path)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 665, in _deactivate_missing_dags
.update({DagModel.is_active: False}, synchronize_session="fetch")
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 4063, in update
update_op.exec_()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1696, in exec_
self._do_pre_synchronize()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1806, in _do_pre_synchronize
select_stmt, mapper=self.mapper, params=query._params
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1295, in execute
return self._connection_for_bind(bind, close_with_result=True).execute(
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1151, in _connection_for_bind
engine, execution_options
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2340, in _wrap_pool_connect
e, dialect, self
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
(Background on this error at: http://sqlalche.me/e/13/e3q8)
[2021-10-28 04:36:17,379] {scheduler_job.py:161} INFO - Exiting gracefully upon receiving signal 15
[2021-10-28 04:36:18,382] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 31
[2021-10-28 04:36:18,889] {process_utils.py:66} INFO - Process psutil.Process(pid=956, status='terminated', started='04:36:17') (956) terminated with exit code None
[2021-10-28 04:36:19,108] {process_utils.py:212} INFO - Waiting up to 5 seconds for processes to exit...
[2021-10-28 04:36:19,142] {process_utils.py:66} INFO - Process psutil.Process(pid=31, status='terminated', exitcode=0, started='04:31:30') (31) terminated with exit code 0
[2021-10-28 04:36:19,143] {process_utils.py:66} INFO - Process psutil.Process(pid=957, status='terminated', started='04:36:17') (957) terminated with exit code None
[2021-10-28 04:36:19,143] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 31
[2021-10-28 04:36:19,144] {scheduler_job.py:614} INFO - Exited execute loop
### What you expected to happen
Upgraded from 2.1.4 to 2.2.0. Noticed the new triggerer pod has restarted a few times and scheduler pods has restarted the same number of times. I don't think any dag execution have occurred. Don't believe there was any issues with 2.1.x version like this that I noticed.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #19275: Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #19275:
URL: https://github.com/apache/airflow/issues/19275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #19275: Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19275:
URL: https://github.com/apache/airflow/issues/19275#issuecomment-953662228
Please update python version. Fixed by https://github.com/apache/airflow/pull/18926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org