You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/28 04:43:58 UTC

[GitHub] [airflow] kn6405 opened a new issue #19275: Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'

kn6405 opened a new issue #19275:
URL: https://github.com/apache/airflow/issues/19275


   ### Apache Airflow version
   
   2.2.0 (latest released)
   
   ### Operating System
   
   Docker Image
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   1.2.0 helm chart and 2.2.0 Airflow
   
   ### What happened
   
   I see scheduler pods and new triggerer pods have restarts. Just upgraded from 2.1.4 to 2.2.0.  
   
   In the triggerer pod logs: 
   
   ____________ _____________
   ____ |__( )_________ __/__ /________ __
   ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
   ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
   _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
   [2021-10-28 04:31:26,124] {triggerer_job.py:101} INFO - Starting the triggerer
   [2021-10-28 04:36:16,431] {triggerer_job.py:111} INFO - Waiting for triggers to clean up
   [2021-10-28 04:36:16,431] {triggerer_job.py:117} INFO - Exited trigger loop
   Exception in thread Thread-1:
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
   self.run()
   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/triggerer_job.py", line 231, in run
   asyncio.run(self.arun()) # pylint: disable=no-member
   AttributeError: module 'asyncio' has no attribute 'run'
   
   Scheduler logs: 
   
   
     ____________       _____________
    ____    |__( )_________  __/__  /________      __
   ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
   ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
    _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
   [2021-10-28 04:31:31,031] {scheduler_job.py:555} INFO - Starting the scheduler
   [2021-10-28 04:31:31,031] {scheduler_job.py:560} INFO - Processing each file at most -1 times
   [2021-10-28 04:31:31,258] {manager.py:163} INFO - Launched DagFileProcessorManager with pid: 31
   [2021-10-28 04:31:31,264] {scheduler_job.py:1066} INFO - Resetting orphaned tasks for active dag runs
   [2021-10-28 04:31:31,272] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
   [2021-10-28 04:31:31,276] {settings.py:462} INFO - Loaded airflow_local_settings from /opt/airflow/config/airflow_local_settings.py .
   Process DagFileProcessor322-Process:
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
       return fn()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
       return _ConnectionFairy._checkout(self)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
       fairy = _ConnectionRecord.checkout(pool)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
       rec = pool._do_get()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
       self._dec_overflow()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
       with_traceback=exc_tb,
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
       return self._create_connection()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
       return _ConnectionRecord(self)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
       self.__connect(first_connect_check=True)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
       pool.logger.debug("Error on connect(): %s", e)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
       with_traceback=exc_tb,
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
       connection = pool._invoke_creator(self)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
       return dialect.connect(*cargs, **cparams)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
       return self.dbapi.connect(*cargs, **cparams)
     File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   psycopg2.OperationalError: server closed the connection unexpectedly
   	This probably means the server terminated abnormally
   	before or while processing the request.
   
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
       self._target(*self._args, **self._kwargs)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 168, in _run_file_processor
       callback_requests=callback_requests,
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 629, in process_file
       self._deactivate_missing_dags(session, dagbag, file_path)
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 665, in _deactivate_missing_dags
       .update({DagModel.is_active: False}, synchronize_session="fetch")
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 4063, in update
       update_op.exec_()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1696, in exec_
       self._do_pre_synchronize()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1806, in _do_pre_synchronize
       select_stmt, mapper=self.mapper, params=query._params
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1295, in execute
       return self._connection_for_bind(bind, close_with_result=True).execute(
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1151, in _connection_for_bind
       engine, execution_options
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
       conn = bind._contextual_connect()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
       self._wrap_pool_connect(self.pool.connect, None),
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2340, in _wrap_pool_connect
       e, dialect, self
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _handle_dbapi_exception_noconnection
       sqlalchemy_exception, with_traceback=exc_info[2], from_=e
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
       return fn()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
       return _ConnectionFairy._checkout(self)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
       fairy = _ConnectionRecord.checkout(pool)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
       rec = pool._do_get()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
       self._dec_overflow()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
       with_traceback=exc_tb,
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
       return self._create_connection()
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
       return _ConnectionRecord(self)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
       self.__connect(first_connect_check=True)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
       pool.logger.debug("Error on connect(): %s", e)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
       with_traceback=exc_tb,
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
       connection = pool._invoke_creator(self)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
       return dialect.connect(*cargs, **cparams)
     File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
       return self.dbapi.connect(*cargs, **cparams)
     File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
   	This probably means the server terminated abnormally
   	before or while processing the request.
   
   (Background on this error at: http://sqlalche.me/e/13/e3q8)
   [2021-10-28 04:36:17,379] {scheduler_job.py:161} INFO - Exiting gracefully upon receiving signal 15
   [2021-10-28 04:36:18,382] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 31
   [2021-10-28 04:36:18,889] {process_utils.py:66} INFO - Process psutil.Process(pid=956, status='terminated', started='04:36:17') (956) terminated with exit code None
   [2021-10-28 04:36:19,108] {process_utils.py:212} INFO - Waiting up to 5 seconds for processes to exit...
   [2021-10-28 04:36:19,142] {process_utils.py:66} INFO - Process psutil.Process(pid=31, status='terminated', exitcode=0, started='04:31:30') (31) terminated with exit code 0
   [2021-10-28 04:36:19,143] {process_utils.py:66} INFO - Process psutil.Process(pid=957, status='terminated', started='04:36:17') (957) terminated with exit code None
   [2021-10-28 04:36:19,143] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 31
   [2021-10-28 04:36:19,144] {scheduler_job.py:614} INFO - Exited execute loop
   
   
   ### What you expected to happen
   
   Upgraded from 2.1.4 to 2.2.0. Noticed the new triggerer pod has restarted a few times and scheduler pods has restarted the same number of times. I don't think any dag execution have occurred. Don't believe there was any issues with 2.1.x version like this that I noticed.
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj closed issue #19275: Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'

Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #19275:
URL: https://github.com/apache/airflow/issues/19275


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #19275: Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19275:
URL: https://github.com/apache/airflow/issues/19275#issuecomment-953662228


   Please update python version. Fixed by https://github.com/apache/airflow/pull/18926


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org