You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/18 11:58:10 UTC
[GitHub] [airflow] ashb opened a new pull request #19677: Add back-compat to db migrations for helm chart < 1.4
ashb opened a new pull request #19677:
URL: https://github.com/apache/airflow/pull/19677
Prior to 1.4 of the helm-chart we "inlined"/hard-coded the
wait-for-migrations command (as a `python -c` command) and that version
didn't correctly initialize the alembic context, meaning that anything
triggering a context.get_bind() at migration version import time caused
an error.
This fixes that problem by:
- Making StringID and TIMESTAMP lazy objects, so that just importing
them at the top level doesn't trigger the problem
- Stop calling it at the top level where it doesn't matter -- we have a
few migrations that create a "copy" of TaskInstance/DagRun model to do
an update, and those have been changed to
a. just use `String()` (as the exact type doesn't matter for UDPATE)
b. to only define the PK+columns being updated, as that is enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972943507
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972819052
```
Monitoring tests: Always API Core Other CLI Providers WWW Integration MySQL
Output for successful API
Output for successful Always
Output for successful CLI
Output for successful Core
Output for successful Integration
Output for successful MySQL
Output for successful Other
Output for successful Providers
Output for successful WWW
Error: Process completed with exit code 1.
```
Ummmmmmm?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972940079
Right, this one green apart from the dag id issue fixed in another PR.
Merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972799855
(Plus this way I get to delete another 50 lines of code, bringing me closer to my goal of net negative LoC in Airflow 😁 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972842326
Yeah, I was just confused by the summary. Fixed that now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972824070
Fix for ci runners incoming so they actually fail!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972821350
```
INFO [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 33, in <module>
sys.exit(load_entry_point('apache-airflow', 'console_scripts', 'airflow')())
File "/opt/airflow/airflow/__main__.py", line 48, in main
args.func(args)
File "/opt/airflow/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/airflow/airflow/cli/commands/db_command.py", line 39, in resetdb
db.resetdb()
File "/opt/airflow/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 930, in resetdb
initdb(session=session)
File "/opt/airflow/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 592, in initdb
upgradedb(session=session)
File "/opt/airflow/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 915, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/airflow/migrations/env.py", line 107, in <module>
run_migrations_online()
File "/opt/airflow/airflow/migrations/env.py", line 101, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
step.migration_fn(**kw)
File "/opt/airflow/airflow/migrations/versions/0a2a5b66e19d_add_task_reschedule_table.py", line 53, in upgrade
sa.Column('execution_date', timestamp, nullable=False, server_default=None),
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 1506, in __init__
super(Column, self).__init__(name, type_)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 4144, in __init__
self.type = type_api.to_instance(type_)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/type_api.py", line 1470, in to_instance
return typeobj(*arg, **kw)
TypeError: 'TIMESTAMP' object is not callable
Error: check_environment returned 1. Exiting.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-973024061
Good one. Much better than trying to workaround iti in import :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #19677:
URL: https://github.com/apache/airflow/pull/19677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #19677: Add back-compat to db migrations for helm chart < 1.4
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #19677:
URL: https://github.com/apache/airflow/pull/19677#issuecomment-972821350
```
INFO [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 33, in <module>
sys.exit(load_entry_point('apache-airflow', 'console_scripts', 'airflow')())
File "/opt/airflow/airflow/__main__.py", line 48, in main
args.func(args)
File "/opt/airflow/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/airflow/airflow/cli/commands/db_command.py", line 39, in resetdb
db.resetdb()
File "/opt/airflow/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 930, in resetdb
initdb(session=session)
File "/opt/airflow/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 592, in initdb
upgradedb(session=session)
File "/opt/airflow/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 915, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/airflow/migrations/env.py", line 107, in <module>
run_migrations_online()
File "/opt/airflow/airflow/migrations/env.py", line 101, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
step.migration_fn(**kw)
File "/opt/airflow/airflow/migrations/versions/0a2a5b66e19d_add_task_reschedule_table.py", line 53, in upgrade
sa.Column('execution_date', timestamp, nullable=False, server_default=None),
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 1506, in __init__
super(Column, self).__init__(name, type_)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 4144, in __init__
self.type = type_api.to_instance(type_)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/type_api.py", line 1470, in to_instance
return typeobj(*arg, **kw)
TypeError: 'TIMESTAMP' object is not callable
Error: check_environment returned 1. Exiting.
```
https://github.com/apache/airflow/runs/4250937621?check_suite_focus=true#step:6:497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org