You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "error691 (via GitHub)" <gi...@apache.org> on 2023/02/09 15:14:48 UTC
[GitHub] [superset] error691 opened a new issue, #23053: superset db upgrade failing on migration from 1.4.2 to 2.0.1
error691 opened a new issue, #23053:
URL: https://github.com/apache/superset/issues/23053
Hi,
I'm trying to upgrade superset from 1.4.2. After `superset db upgrade` I faced with migration error.
```
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "user_id" violates not-null constraint
DETAIL: Failing row contains (4405, null).
[SQL: INSERT INTO sl_dataset_users (dataset_id, user_id) SELECT sl_datasets.id AS dataset_id, sqlatable_user.user_id
FROM sqlatable_user JOIN tables ON tables.id = sqlatable_user.table_id JOIN sl_datasets ON sl_datasets.uuid = tables.uuid]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
```
#### How to reproduce the bug
Upgrade from 1.4.2 to 2.0.1
superset db upgrade
### Expected results
Successful migration.
### Actual results
`INFO [alembic.runtime.migration] Running upgrade cecc6bf46990 -> ad07e4fdbaba, rm_time_range_endpoints_from_qc_3
slices updated with no time_range_endpoints: 4264
INFO [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2
>> Copy 220 physical tables to sl_tables...
>> Copy 4,621 SqlaTable to sl_datasets...
Copy dataset owners...
Traceback (most recent call last):
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "user_id" violates not-null constraint
DETAIL: Failing row contains (4405, null).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/ss_venv/bin/superset", line 8, in <module>
sys.exit(superset())
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/flask/cli.py", line 567, in main
return super().main(*args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/flask/cli.py", line 407, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/flask_migrate/cli.py", line 150, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/ss_venv/lib64/python3.8/site-packages/flask_migrate/__init__.py", line 111, in wrapped
f(*args, **kwargs)
File "/ss_venv/lib64/python3.8/site-packages/flask_migrate/__init__.py", line 200, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/ss_venv/lib64/python3.8/site-packages/alembic/command.py", line 378, in upgrade
script.run_env()
File "/ss_venv/lib64/python3.8/site-packages/alembic/script/base.py", line 569, in run_env
util.load_python_file(self.dir, "env.py")
File "/ss_venv/lib64/python3.8/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/ss_venv/lib64/python3.8/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/ss_venv/lib64/python3.8/site-packages/superset/extensions/../migrations/env.py", line 126, in <module>
run_migrations_online()
File "/ss_venv/lib64/python3.8/site-packages/superset/extensions/../migrations/env.py", line 118, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/ss_venv/lib64/python3.8/site-packages/alembic/runtime/environment.py", line 867, in run_migrations
self.get_context().run_migrations(**kw)
File "/ss_venv/lib64/python3.8/site-packages/alembic/runtime/migration.py", line 624, in run_migrations
step.migration_fn(**kw)
File "/ss_venv/lib64/python3.8/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 876, in upgrade
copy_datasets(session)
File "/ss_venv/lib64/python3.8/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 379, in copy_datasets
insert_from_select(
File "/ss_venv/lib64/python3.8/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 102, in insert_from_select
return op.execute(query)
File "<string>", line 8, in execute
File "<string>", line 3, in execute
File "/ss_venv/lib64/python3.8/site-packages/alembic/operations/ops.py", line 2416, in execute
return operations.invoke(op)
File "/ss_venv/lib64/python3.8/site-packages/alembic/operations/base.py", line 401, in invoke
return fn(self, operation)
File "/ss_venv/lib64/python3.8/site-packages/alembic/operations/toimpl.py", line 207, in execute_sql
operations.migration_context.impl.execute(
File "/ss_venv/lib64/python3.8/site-packages/alembic/ddl/impl.py", line 202, in execute
self._exec(sql, execution_options)
File "/ss_venv/lib64/python3.8/site-packages/alembic/ddl/impl.py", line 193, in _exec
return conn.execute( # type: ignore[call-overload]
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self._handle_dbapi_exception(
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
util.raise_(
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/ss_venv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "user_id" violates not-null constraint
DETAIL: Failing row contains (4405, null).
[SQL: INSERT INTO sl_dataset_users (dataset_id, user_id) SELECT sl_datasets.id AS dataset_id, sqlatable_user.user_id
FROM sqlatable_user JOIN tables ON tables.id = sqlatable_user.table_id JOIN sl_datasets ON sl_datasets.uuid = tables.uuid]
(Background on this error at: http://sqlalche.me/e/13/gkpj)`
### Environment
- Docker env
- Base image: `centos8`
- superset version: `2.0.1`
- python version: `3.8.8`
- Postgres version: `11.1`
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [x] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
### Additional context
After the migration fail tables sl_dataset_users and sl_datasets are absent in DB. I'm not able to reproduce the select
`SELECT sl_datasets.id AS dataset_id, sqlatable_user.user_id
FROM sqlatable_user JOIN tables ON tables.id = sqlatable_user.table_id JOIN sl_datasets ON sl_datasets.uuid = tables.uuid`
A clean installation runs without any problems.
It seams like the issue https://github.com/apache/superset/issues/20824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] josechudev commented on issue #23053: superset db upgrade failing on migration from 1.4.2 to 2.0.1
Posted by "josechudev (via GitHub)" <gi...@apache.org>.
josechudev commented on issue #23053:
URL: https://github.com/apache/superset/issues/23053#issuecomment-1437388118
Hi @error691, did you manage to solve this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] error691 commented on issue #23053: superset db upgrade failing on migration from 1.4.2 to 2.0.1
Posted by "error691 (via GitHub)" <gi...@apache.org>.
error691 commented on issue #23053:
URL: https://github.com/apache/superset/issues/23053#issuecomment-1438046761
Hi,
Yes, the issue was solved. Some datasets have more than one owner and not all of owners were existed.
Empty records were deleted - `delete from sqlatable_user where user_id is null;`
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] error691 closed issue #23053: superset db upgrade failing on migration from 1.4.2 to 2.0.1
Posted by "error691 (via GitHub)" <gi...@apache.org>.
error691 closed issue #23053: superset db upgrade failing on migration from 1.4.2 to 2.0.1
URL: https://github.com/apache/superset/issues/23053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org