You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/05 18:20:57 UTC
[GitHub] [airflow] thsubramani opened a new issue #20695: Airflow 2.2.3 with Schedulers HA Failing on postgres 13
thsubramani opened a new issue #20695:
URL: https://github.com/apache/airflow/issues/20695
### Apache Airflow version
2.2.3 (latest released)
### What happened
airflow scheduler POD with HA connecting to same postgres DB connecting with beloe error even though use_row_level_locking=true dy defualt/explicitly setting as env.
airflow-c-qa4-64956645c-tr4zm airflow-scheduler [2022-01-05 17:18:26,109] {scheduler_job.py:721} INFO - Exited execute loop
airflow-c-qa4-64956645c-tr4zm airflow-scheduler Traceback (most recent call last):
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor, statement, parameters, context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor.execute(statement, parameters)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler psycopg2.errors.LockNotAvailable: could not obtain lock on row in relation "slot_pool"
airflow-c-qa4-64956645c-tr4zm airflow-scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler The above exception was the direct cause of the following exception:
airflow-c-qa4-64956645c-tr4zm airflow-scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler Traceback (most recent call last):
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/bin/airflow", line 8, in <module>
airflow-c-qa4-64956645c-tr4zm airflow-scheduler sys.exit(main())
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
airflow-c-qa4-64956645c-tr4zm airflow-scheduler args.func(args)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return func(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return f(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/scheduler_command.py", line 70, in scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler job.run()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 245, in run
airflow-c-qa4-64956645c-tr4zm airflow-scheduler self._execute()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 694, in _execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler self._run_scheduler_loop()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 787, in _run_scheduler_loop
airflow-c-qa4-64956645c-tr4zm airflow-scheduler num_queued_tis = self._do_scheduling(session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 926, in _do_scheduling
airflow-c-qa4-64956645c-tr4zm airflow-scheduler num_queued_tis = self._critical_section_execute_task_instances(session=session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 550, in _critical_section_execute_task_instances
airflow-c-qa4-64956645c-tr4zm airflow-scheduler queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 67, in wrapper
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return func(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 307, in _executable_task_instances_to_queued
airflow-c-qa4-64956645c-tr4zm airflow-scheduler pools = models.Pool.slots_stats(lock_rows=True, session=session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 67, in wrapper
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return func(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/pool.py", line 107, in slots_stats
airflow-c-qa4-64956645c-tr4zm airflow-scheduler pool_rows: Iterable[Tuple[str, int]] = query.all()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return list(self)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return self._execute_and_instances(context)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
airflow-c-qa4-64956645c-tr4zm airflow-scheduler result = conn.execute(querycontext.statement, self._params)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return meth(self, multiparams, params)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return connection._execute_clauseelement(self, multiparams, params)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
airflow-c-qa4-64956645c-tr4zm airflow-scheduler distilled_params,
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler e, statement, parameters, cursor, context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1515, in _handle_dbapi_exception
airflow-c-qa4-64956645c-tr4zm airflow-scheduler util.raise_(newraise, with_traceback=exc_info[2], from_=e)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
airflow-c-qa4-64956645c-tr4zm airflow-scheduler raise exception
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor, statement, parameters, context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor.execute(statement, parameters)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler AttributeError: 'PGExecutionContext_psycopg2' object has no attribute '_stan_scope'
### What you expected to happen
airflow scheduler should not crash
### How to reproduce
bring uo airflow scheduler 2 pods connecting to same postgres DB
### Operating System
docker
### Versions of Apache Airflow Providers
2.1.0,2.1.1,2.1.3
### Deployment
Other Docker-based deployment
### Deployment details
k8s
### Anything else
nothing much
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20695: Airflow 2.2.3 with Schedulers HA Failing on postgres 13
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20695:
URL: https://github.com/apache/airflow/issues/20695#issuecomment-1019370098
I think this problem is because you are not using proper installation method. Old version of postgres libraries apparently exhibit this problem but later version should be fine (at least that's why quick search returned)
Follow the instructions here: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20695: Airflow 2.2.3 with Schedulers HA Failing on postgres 13
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20695:
URL: https://github.com/apache/airflow/issues/20695#issuecomment-1005968973
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org