You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/22 05:42:00 UTC
[GitHub] [airflow] DreamyWen opened a new issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
DreamyWen opened a new issue #13809:
URL: https://github.com/apache/airflow/issues/13809
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**:
2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
Python 3.7.3
Linux Version
uname -r
3.10.0-693.21.1.el7.x86_64
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
mysql instance version
5.6.26-log
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
<!-- (please include exact error messages if you can) -->
i use command to install airflow
`
pip install apache-airflow
pip install apache-airflow[mysql]
`
then change the airflow.cfg
`sql_alchemy_conn = mysql+pymysql://airflow:xxx@mydomain:3653/airflow`
then execute the command
`airflow db init`
the error occured
INFO [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> d38e04c12aa2, add serialized_dag table
Traceback (most recent call last):
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1278, in _execute_context
cursor, statement, parameters, context
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1305, 'FUNCTION airflow.JSON_VALID does not exist')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data/anaconda3/bin/airflow", line 10, in <module>
sys.exit(main())
File "/data/anaconda3/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/data/anaconda3/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/data/anaconda3/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
db.initdb()
File "/data/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
upgradedb()
File "/data/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py", line 688, in upgradedb
command.upgrade(config, 'heads')
File "/data/anaconda3/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
script.run_env()
File "/data/anaconda3/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
util.load_python_file(self.dir, "env.py")
File "/data/anaconda3/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/data/anaconda3/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/data/anaconda3/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/data/anaconda3/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/data/anaconda3/lib/python3.7/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
self.get_context().run_migrations(**kw)
File "/data/anaconda3/lib/python3.7/site-packages/alembic/runtime/migration.py", line 522, in run_migrations
step.migration_fn(**kw)
File "/data/anaconda3/lib/python3.7/site-packages/airflow/migrations/versions/d38e04c12aa2_add_serialized_dag_table.py", line 46, in upgrade
conn.execute("SELECT JSON_VALID(1)").fetchone()
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1006, in execute
return self._execute_text(object_, multiparams, params)
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1181, in _execute_text
parameters,
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1318, in _execute_context
e, statement, parameters, cursor, context
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1512, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1278, in _execute_context
cursor, statement, parameters, context
File "/data/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/data/anaconda3/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1305, 'FUNCTION airflow.JSON_VALID does not exist')
[SQL: SELECT JSON_VALID(1)]
(Background on this error at: http://sqlalche.me/e/13/2j85)
**What you expected to happen**:
the db should init success
<!-- What do you think went wrong? -->
i use mysql version 5.6.26, which is deployed in company env, it's hard to upgrade to 5.7+.
i wonder is there a workarond to use airflow 2.0.0 with mysql 5.6.26, can i bypass the JSON_VALID function?
**How to reproduce it**:
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
install airflow version 2.0.0 with mysql instance 5.6.26 then init db should reproduce it.
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13809:
URL: https://github.com/apache/airflow/issues/13809#issuecomment-764548237
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13809:
URL: https://github.com/apache/airflow/issues/13809#issuecomment-764626601
Airflow 2.0 does not work with MySQL 5.6: https://github.com/apache/airflow/blob/master/README.md#requirements. You need to upgrade to 5.7 or even better to MySQL 8 as it has better scalability features.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13809:
URL: https://github.com/apache/airflow/issues/13809
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13809:
URL: https://github.com/apache/airflow/issues/13809
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13809:
URL: https://github.com/apache/airflow/issues/13809#issuecomment-764548237
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13809: FUNCTION airflow.JSON_VALID with airflow version 2.0.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13809:
URL: https://github.com/apache/airflow/issues/13809#issuecomment-764626601
Airflow 2.0 does not work with MySQL 5.6: https://github.com/apache/airflow/blob/master/README.md#requirements. You need to upgrade to 5.7 or even better to MySQL 8 as it has better scalability features.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org