You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Smith Mathieu (JIRA)" <ji...@apache.org> on 2018/09/11 00:58:00 UTC
[jira] [Updated] (AIRFLOW-3036) Upgrading to Airflow 1.10 not
possible using GCP Cloud SQL for MYSQL
[ https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Smith Mathieu updated AIRFLOW-3036:
-----------------------------------
Description:
The upgrade path to airflow 1.10 seems impossible for users of MySQL in Google's Cloud SQL service given new mysql requirements for 1.10.
When executing "airflow upgradedb"
```
INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 32, in <module>
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, in initdb
db_utils.initdb(settings.RBAC)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, in initdb
upgradedb()
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
script.run_env()
File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
module_id, path).load_module(module_id)
File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
File "<frozen importlib._bootstrap_external>", line 823, in load_module
File "<frozen importlib._bootstrap_external>", line 682, in load_module
File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 91, in <module>
run_migrations_online()
File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 86, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 807, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 321, in run_migrations
step.migration_fn(**kw)
File "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", line 46, in upgrade
raise Exception("Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql")
Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
```
Reading documentation for upgrading to airflow 1.10, it seems the requirement for explicit_defaults_for_timestamp=1 was intentional.
However, MySQL on Google Cloud SQL does not support configuring this variable and it is off by default. Users of MySQL and Cloud SQL do not have an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
In GCP, Cloud SQL is _the_ hosted MySQL solution.
[https://cloud.google.com/sql/docs/mysql/flags]
was:
The upgrade path to airflow 1.10 seems impossible for users of MySQL in Google's Cloud SQL service given new mysql requirements for 1.10.
When executing "airflow upgradedb"
```
INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 32, in <module>
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, in initdb
db_utils.initdb(settings.RBAC)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, in initdb
upgradedb()
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
script.run_env()
File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
module_id, path).load_module(module_id)
File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
File "<frozen importlib._bootstrap_external>", line 823, in load_module
File "<frozen importlib._bootstrap_external>", line 682, in load_module
File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 91, in <module>
run_migrations_online()
File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 86, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 807, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 321, in run_migrations
step.migration_fn(**kw)
File "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", line 46, in upgrade
raise Exception("Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql")
Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
```
Reading documentation for upgrading to airflow 1.10, it seems the requirement for explicit_defaults_for_timestamp=1 was intentional.
However, MySQL on Google Cloud SQL does not support configuring this variable and it is off by default. Users of MySQL and Cloud SQL do not have an upgrade path to 1.10
In GCP, Cloud SQL is _the_ hosted MySQL solution.
https://cloud.google.com/sql/docs/mysql/flags
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
> --------------------------------------------------------------------
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
> Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow 1.10 on Debian Stretch, Google Cloud SQL MySQL
> Reporter: Smith Mathieu
> Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in <module>
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002, in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
> module_id, path).load_module(module_id)
> File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
> File "<frozen importlib._bootstrap_external>", line 823, in load_module
> File "<frozen importlib._bootstrap_external>", line 682, in load_module
> File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
> File "<frozen importlib._bootstrap>", line 684, in _load
> File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
> File "<frozen importlib._bootstrap_external>", line 678, in exec_module
> File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 91, in <module>
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 86, in run_migrations_online
> context.run_migrations()
> File "<string>", line 8, in run_migrations
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 321, in run_migrations
> step.migration_fn(**kw)
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this variable and it is off by default. Users of MySQL and Cloud SQL do not have an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)