You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aran3 (via GitHub)" <gi...@apache.org> on 2023/02/22 12:58:37 UTC
[GitHub] [airflow] aran3 opened a new issue, #29693: airflow db downgrade --show-sql-only raised AttributeError
aran3 opened a new issue, #29693:
URL: https://github.com/apache/airflow/issues/29693
### Apache Airflow version
2.5.1
### What happened
When running a dry-run of a downgrade from airflow 2.5.1 to 1.10.15, the operation ends with an AttributeError raised, accessing `MockConnection.close`.
This seems related to what is explain in https://github.com/sqlalchemy/alembic/issues/740 - the migration does a reflection operation.
Command that was executed:
`airflow db downgrade --to-revision 03afc6b6f902 --show-sql-only`
Log end:
```text
2023-02-22T12:10:27.375Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 128, in _init_engine
2023-02-22T12:10:27.375Z engine.connect().close()
2023-02-22T12:10:27.375Z AttributeError: 'MockConnection' object has no attribute 'close'
```
<details><summary>Full log details</summary>
<p>
```text
2023-02-22T12:10:19.196Z Generating sql for downgrade -- downgrade commands will *not* be submitted.
2023-02-22T12:10:19.196Z [~[34m2023-02-22 07:10:19,195~[0m] {~[34mdb.py:~[0m1649} INFO~[0m - Attempting downgrade to revision 03afc6b6f902~[0m
2023-02-22T12:10:19.219Z [~[34m2023-02-22 07:10:19,219~[0m] {~[34mdb.py:~[0m1654} WARNING~[0m - Generating sql scripts for manual migration.~[0m
2023-02-22T12:10:19.226Z [~[34m2023-02-22 07:10:19,225~[0m] {~[34mmigration.py:~[0m205} INFO~[0m - Context impl PostgresqlImpl.~[0m
2023-02-22T12:10:19.226Z [~[34m2023-02-22 07:10:19,226~[0m] {~[34mmigration.py:~[0m208} INFO~[0m - Will assume transactional DDL.~[0m
2023-02-22T12:10:21.134Z BEGIN;
2023-02-22T12:10:21.134Z
2023-02-22T12:10:21.356Z -- Running downgrade 290244fb8b83 -> 1986afd32c1b
...
2023-02-22T12:10:27.359Z
2023-02-22T12:10:27.359Z -- Running downgrade bef4f3d11e8b -> e1a11ece99cc
2023-02-22T12:10:27.359Z
2023-02-22T12:10:27.374Z Traceback (most recent call last):
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/bin/../dependencies/.bin/airflow", line 8, in <module>
2023-02-22T12:10:27.374Z sys.exit(main())
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/__main__.py", line 39, in main
2023-02-22T12:10:27.374Z args.func(args)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/cli/cli_parser.py", line 52, in command
2023-02-22T12:10:27.374Z return func(*args, **kwargs)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/utils/cli.py", line 108, in wrapper
2023-02-22T12:10:27.374Z return f(*args, **kwargs)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 134, in downgrade
2023-02-22T12:10:27.374Z db.downgrade(to_revision=to_revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/utils/session.py", line 75, in wrapper
2023-02-22T12:10:27.374Z return func(*args, session=session, **kwargs)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/utils/db.py", line 1658, in downgrade
2023-02-22T12:10:27.374Z _offline_migration(command.downgrade, config=config, revision=revision_range)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/utils/db.py", line 1447, in _offline_migration
2023-02-22T12:10:27.374Z migration_func(config, revision, sql=True)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/alembic/command.py", line 424, in downgrade
2023-02-22T12:10:27.374Z script.run_env()
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
2023-02-22T12:10:27.374Z util.load_python_file(self.dir, "env.py")
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
2023-02-22T12:10:27.374Z module = load_module_py(module_id, path)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
2023-02-22T12:10:27.374Z spec.loader.exec_module(module) # type: ignore
2023-02-22T12:10:27.374Z File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2023-02-22T12:10:27.374Z File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/migrations/env.py", line 115, in <module>
2023-02-22T12:10:27.374Z run_migrations_offline()
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/migrations/env.py", line 84, in run_migrations_offline
2023-02-22T12:10:27.374Z context.run_migrations()
2023-02-22T12:10:27.374Z File "<string>", line 8, in run_migrations
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/alembic/runtime/environment.py", line 867, in run_migrations
2023-02-22T12:10:27.374Z self.get_context().run_migrations(**kw)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/alembic/runtime/migration.py", line 624, in run_migrations
2023-02-22T12:10:27.374Z step.migration_fn(**kw)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/airflow/migrations/versions/0068_2_0_0_drop_kuberesourceversion_and_.py", line 58, in downgrade
2023-02-22T12:10:27.374Z inspector = inspect(conn)
2023-02-22T12:10:27.374Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/sqlalchemy/inspection.py", line 64, in inspect
2023-02-22T12:10:27.374Z ret = reg(subject)
2023-02-22T12:10:27.375Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 178, in _connectable_insp
2023-02-22T12:10:27.375Z return Inspector._construct(Inspector._init_legacy, bind)
2023-02-22T12:10:27.375Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 117, in _construct
2023-02-22T12:10:27.375Z init(self, bind)
2023-02-22T12:10:27.375Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 124, in _init_legacy
2023-02-22T12:10:27.375Z self._init_engine(bind)
2023-02-22T12:10:27.375Z File "/bb/bin/package/b/btca-airflow-venv/0.9.0+devise/libexec/btca-airflow-venv/python/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 128, in _init_engine
2023-02-22T12:10:27.375Z engine.connect().close()
2023-02-22T12:10:27.375Z AttributeError: 'MockConnection' object has no attribute 'close'
```
</p>
</details>
As the error occurs while running dowgrade from `bef4f3d11e8b -> e1a11ece99cc`, the next migration steps are not printed.
### What you think should happen instead
I am aware the `db downgrade` is not supported to downgrade to older airflow versions officially, but as the only reason this fails is the "close" access on the member, maybe this error can be caught and ignored.
### How to reproduce
Upgrade from older airflow to 2.5.1 (probably any 2.X that supports the `db downgrade` command, as the failure is related to a revision in 2.0.0), then run
`airflow db downgrade --to-revision 03afc6b6f902 --show-sql-only`
### Operating System
Red Hat Enterprise Linux Server 7.9
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
PostgreSql DB
Virtual environment created from constraint file for 2.5.1, python3.10
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29693:
URL: https://github.com/apache/airflow/issues/29693#issuecomment-1441663988
I am inclined to close this one as won't fix. @ephraimbuddy ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aran3 commented on issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "aran3 (via GitHub)" <gi...@apache.org>.
aran3 commented on issue #29693:
URL: https://github.com/apache/airflow/issues/29693#issuecomment-1441657864
I was also about to open several issues I found while trying to actually downgrade from 2.5.1 back to 1.10.15, and while I know the `airflow db downgrade` command does not support versions below 2.0.0, I was under the impression that the alembic operations are supposed to be backward compatible.
I also ran such downgrade from 2.1.3 to 1.10.15, several times.
Is it no longer supported to downgrade to 1.10.X at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #29693:
URL: https://github.com/apache/airflow/issues/29693#issuecomment-1441664286
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #29693:
URL: https://github.com/apache/airflow/issues/29693#issuecomment-1441601843
For postgres, you can run offline migration from 2.0 to the latest and back to 2.0. Others have a version limit where you cannot run offline migration. This is probably documented, but if not, documentation would be greatly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29693:
URL: https://github.com/apache/airflow/issues/29693#issuecomment-1441662328
> I was also about to open several issues I found while trying to actually downgrade from 2.5.1 back to 1.10.15, and while I know the `airflow db downgrade` command does not support versions below 2.0.0, I was under the impression that the alembic operations are supposed to be backward compatible. I also ran such downgrade from 2.1.3 to 1.10.15, several times. Is it no longer supported to downgrade to 1.10.X at all?
It has never been possible. It might work accidentally but. The only way to go back to 1.10* was to restore from backup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29693: airflow db downgrade --show-sql-only raised AttributeError
URL: https://github.com/apache/airflow/issues/29693
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cynhash commented on issue #29693: airflow db downgrade --show-sql-only raised AttributeError
Posted by "cynhash (via GitHub)" <gi...@apache.org>.
cynhash commented on issue #29693:
URL: https://github.com/apache/airflow/issues/29693#issuecomment-1441506529
Hi, I'm new to open source and would love to work on this issue. Can I take this up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org