You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/22 07:17:19 UTC
[GitHub] [airflow] jicki opened a new issue #14354: airflow version 2.0.1 === airflow db upgrade error
jicki opened a new issue #14354:
URL: https://github.com/apache/airflow/issues/14354
```
[2021-02-22 06:17:59,228] {db.py:674} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
INFO [alembic.runtime.migration] Running upgrade 03afc6b6f902 -> cf5dc11e79ad, drop_user_and_chart
INFO [alembic.runtime.migration] Running upgrade cf5dc11e79ad -> bbf4a7ad0465, Remove id column from xcom
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 255, in execute
self.errorhandler(self, exc, value)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 252, in execute
res = self._query(query)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 378, in _query
db.query(q)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1071, 'Specified key was too long; max key length is 3072 bytes')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/db_command.py", line 48, in upgradedb
db.upgradedb()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 684, in upgradedb
command.upgrade(config, 'heads')
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/command.py", line 294, in upgrade
script.run_env()
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/script/base.py", line 481, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/util/compat.py", line 182, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 560, in run_migrations
step.migration_fn(**kw)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/migrations/versions/bbf4a7ad0465_remove_id_column_from_xcom.py", line 48, in upgrade
bop.create_primary_key('pk_xcom', ['dag_id', 'task_id', 'key', 'execution_date'])
File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/operations/base.py", line 336, in batch_alter_table
impl.flush()
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/operations/batch.py", line 85, in flush
fn(*arg, **kw)
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 263, in add_constraint
self._exec(schema.AddConstraint(const))
File "/home/airflow/.local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 146, in _exec
return conn.execute(construct, multiparams)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1073, in _execute_ddl
compiled,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 255, in execute
self.errorhandler(self, exc, value)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 252, in execute
res = self._query(query)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 378, in _query
db.query(q)
File "/home/airflow/.local/lib/python3.6/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1071, 'Specified key was too long; max key length is 3072 bytes')
[SQL: ALTER TABLE xcom ADD CONSTRAINT pk_xcom PRIMARY KEY (dag_id, task_id, `key`, execution_date)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14354: airflow version 2.0.1 === airflow db upgrade error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14354:
URL: https://github.com/apache/airflow/issues/14354#issuecomment-783460952
This is because you have MySQL 5.6. You need MySQL 5.7 and above to use Airflow 2.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jicki commented on issue #14354: airflow version 2.0.1 === airflow db upgrade error
Posted by GitBox <gi...@apache.org>.
jicki commented on issue #14354:
URL: https://github.com/apache/airflow/issues/14354#issuecomment-783801258
> This is because you have MySQL 5.6. You need MySQL 5.7 and above to use Airflow 2.0.
MySQL 5.7 Airflow 1.10.13 desc xcom:
```
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 101439446
Server version: 5.7.31-log MySQL Community Server (GPL)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
[MySQL][(none)]> use airflow
Database changed
[MySQL][airflow]> desc xcom;
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| key | varchar(512) | YES | | NULL | |
| value | blob | YES | | NULL | |
| timestamp | timestamp(6) | NO | | NULL | |
| execution_date | timestamp(6) | NO | | NULL | |
| task_id | varchar(250) | NO | | NULL | |
| dag_id | varchar(250) | NO | MUL | NULL | |
+----------------+--------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14354: airflow version 2.0.1 === airflow db upgrade error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14354:
URL: https://github.com/apache/airflow/issues/14354#issuecomment-784011866
Then you probably have `utf8mb4` encoding. In such case you need to specify `sql_engine_collation_for_ids=utf8mb3_general_ci` parameter in Airflow's config before migration. This is due to limitations of MySQL - when utf8mb4 is used, each character takes 4 bytes and it exceed the maximum key size.
We already have a chapter about it in the upcoming documentation (will be released with the next release of Airflow):
https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/set-up-database.rst#setting-up-a-mysql-database
```
In addition, you also should pay particular attention to MySQL's encoding. Although the utf8mb4 character set is more and more popular for MySQL (actually, utf8mb4 becomes default character set in MySQL8.0), using the utf8mb4 encoding requires additional setting in Airflow 2+ (See more details in #7570.). If you use utf8mb4 as character set, you should also set sql_engine_collation_for_ids=utf8mb3_general_ci.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #14354: airflow version 2.0.1 === airflow db upgrade error
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #14354:
URL: https://github.com/apache/airflow/issues/14354#issuecomment-784011866
Then you probably have `utf8mb4` encoding. In such case you need to specify `sql_engine_collation_for_ids=utf8mb3_general_ci` parameter in Airflow's config before migration. This is due to limitations of MySQL - when utf8mb4 is used, each character takes 4 bytes and it exceed the maximum key size.
We already have a chapter about it in the upcoming documentation (will be released with the next release of Airflow):
https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/set-up-database.rst#setting-up-a-mysql-database
```
In addition, you also should pay particular attention to MySQL's encoding. Although the utf8mb4 character set is more
and more popular for MySQL (actually, utf8mb4 becomes default character set in MySQL8.0), using the utf8mb4 encoding
requires additional setting in Airflow 2+ (See more details in #7570.). If you use utf8mb4 as character set, you should also set
sql_engine_collation_for_ids=utf8mb3_general_ci.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #14354: airflow version 2.0.1 === airflow db upgrade error
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #14354:
URL: https://github.com/apache/airflow/issues/14354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14354: airflow version 2.0.1 === airflow db upgrade error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14354:
URL: https://github.com/apache/airflow/issues/14354#issuecomment-783148709
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org