You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/31 13:33:54 UTC
[GitHub] [airflow] jens-scheffler-bosch commented on issue #23512: Random "duplicate key value violates unique constraint" errors when initializing the postgres database
jens-scheffler-bosch commented on issue #23512:
URL: https://github.com/apache/airflow/issues/23512#issuecomment-1297099635
Thanks for the good news. I tried to test in our Azure deployment (using the official Helm chart) and Airflow 2.4.2 and it seems in our deployment it is still showing the same constraint errors. I am a bit confused because the milestoone was now modfied to be Airflow 2.4.3 but above the success is reported on 2.4.2... which release it it assumed to be in?
Our logs still look like attached (don't want to spoil the message thread with hundrets of lines...,):
```
(...)
[2022-10-31 12:59:33 +0000] [41] [INFO] Worker exiting (pid: 41)
[2022-10-31 12:59:33 +0000] [42] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
DETAIL: Key (permission_view_id, role_id)=(3097, 1) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/util.py", line 412, in import_app
app = app(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 167, in cached_app
app = create_app(config=config, testing=testing)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 154, in create_app
sync_appbuilder_roles(flask_app)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 73, in sync_appbuilder_roles
flask_app.appbuilder.sm.sync_roles()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 600, in sync_roles
self.update_admin_permission()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 582, in update_admin_permission
self.get_session.commit()
File "<string>", line 2, in commit
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1428, in commit
self._transaction.commit(_to_root=self.future)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3345, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3485, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3445, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 579, in execute
self.dependency_processor.process_saves(uow, states)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/dependency.py", line 1182, in process_saves
self._run_crud(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/dependency.py", line 1245, in _run_crud
connection.execute(statement, secondary_insert)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
ret = self._execute_context(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
DETAIL: Key (permission_view_id, role_id)=(3097, 1) already exists.
[SQL: INSERT INTO ab_permission_view_role (permission_view_id, role_id) VALUES (%(permission_view_id)s, %(role_id)s) RETURNING ab_permission_view_role.id]
[parameters: {'permission_view_id': 3097, 'role_id': 1}]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
[2022-10-31 12:59:33 +0000] [40] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
DETAIL: Key (permission_view_id, role_id)=(3097, 1) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/util.py", line 412, in import_app
app = app(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 167, in cached_app
app = create_app(config=config, testing=testing)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 154, in create_app
sync_appbuilder_roles(flask_app)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/app.py", line 73, in sync_appbuilder_roles
flask_app.appbuilder.sm.sync_roles()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 600, in sync_roles
self.update_admin_permission()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 582, in update_admin_permission
self.get_session.commit()
File "<string>", line 2, in commit
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1428, in commit
self._transaction.commit(_to_root=self.future)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3345, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3485, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3445, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 579, in execute
self.dependency_processor.process_saves(uow, states)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/dependency.py", line 1182, in process_saves
self._run_crud(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/dependency.py", line 1245, in _run_crud
connection.execute(statement, secondary_insert)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
ret = self._execute_context(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
DETAIL: Key (permission_view_id, role_id)=(3097, 1) already exists.
[SQL: INSERT INTO ab_permission_view_role (permission_view_id, role_id) VALUES (%(permission_view_id)s, %(role_id)s) RETURNING ab_permission_view_role.id]
[parameters: {'permission_view_id': 3097, 'role_id': 1}]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
[2022-10-31 12:59:33 +0000] [42] [INFO] Worker exiting (pid: 42)
[2022-10-31 12:59:33 +0000] [40] [INFO] Worker exiting (pid: 40)
10.10.16.4 - - [31/Oct/2022:12:59:33 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "-"
10.10.16.7 - - [31/Oct/2022:12:59:33 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "-"
10.10.0.241 - - [31/Oct/2022:12:59:34 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "kube-probe/1.23"
10.10.0.241 - - [31/Oct/2022:12:59:34 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "kube-probe/1.23"
10.10.0.241 - - [31/Oct/2022:12:59:34 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "kube-probe/1.23"
10.10.0.241 - - [31/Oct/2022:12:59:34 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "kube-probe/1.23"
10.10.16.4 - - [31/Oct/2022:12:59:35 +0000] "GET /dev-**readacted**/airflow/health HTTP/1.1" 200 141 "-" "-"
[2022-10-31 12:59:35 +0000] [43] [INFO] Worker exiting (pid: 43)
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 209, in run
self.sleep()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 357, in sleep
ready = select.select([self.PIPE[0]], [], [], 1.0)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/__main__.py", line 7, in <module>
run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 229, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 342, in halt
self.stop()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 393, in stop
time.sleep(0.1)
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[2022-10-31T12:59:37.555+0000] {webserver_command.py:252} ERROR - [0 / 0] Some workers seem to have died and gunicorn did not restart them as expected
[2022-10-31T12:59:47.567+0000] {webserver_command.py:264} INFO - [0 / 0] Spawning 1 workers
[2022-10-31T13:01:14.490+0000] {webserver_command.py:420} INFO - Received signal: 15. Closing gunicorn.
```
[webserver-logs.txt](https://github.com/apache/airflow/files/9901473/webserver-logs.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org