You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/25 14:26:53 UTC
[GitHub] [airflow] kaxil opened a new pull request #13892: Make Smart Sensors DB Migration idempotent
kaxil opened a new pull request #13892:
URL: https://github.com/apache/airflow/pull/13892
While developing locally to fix a Scheduler bug, I had to reset DB number of times and this particular
migration caused problems (failed with following error):
```
INFO [alembic.runtime.migration] Running upgrade cf5dc11e79ad -> bbf4a7ad0465, Remove id column from xcom
INFO [alembic.runtime.migration] Running upgrade bbf4a7ad0465 -> b25a55525161, Increase length of pool name
INFO [alembic.runtime.migration] Running upgrade b25a55525161 -> 3c20cacc0044, Add DagRun run_type
INFO [alembic.runtime.migration] Running upgrade 3c20cacc0044 -> 8f966b9c467a, Set conn_type as non-nullable
INFO [alembic.runtime.migration] Running upgrade 8f966b9c467a -> 8d48763f6d53, add unique constraint to conn_id
INFO [alembic.runtime.migration] Running upgrade 8d48763f6d53 -> e38be357a868, Add sensor_instance table
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.DuplicateTable: relation "sensor_instance" already exists
```
Making it idempotent helps us to run it multiple times
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#discussion_r563774796
##########
File path: airflow/migrations/versions/e38be357a868_update_schema_for_smart_sensor.py
##########
@@ -50,6 +51,11 @@ def sa_timestamp(): # noqa: D103
def upgrade(): # noqa: D103
conn = op.get_bind()
+ inspector = Inspector.from_engine(conn)
+ tables = inspector.get_table_names()
+ if "sensor_instance" in tables:
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#issuecomment-766879019
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a change in pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#discussion_r563764874
##########
File path: airflow/migrations/versions/e38be357a868_update_schema_for_smart_sensor.py
##########
@@ -50,6 +51,11 @@ def sa_timestamp(): # noqa: D103
def upgrade(): # noqa: D103
conn = op.get_bind()
+ inspector = Inspector.from_engine(conn)
+ tables = inspector.get_table_names()
+ if "sensor_instance" in tables:
Review comment:
nit: good to use single quotes to keep consistent with the whole script? The same for the change in `downgrade()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG merged pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
XD-DENG merged pull request #13892:
URL: https://github.com/apache/airflow/pull/13892
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a change in pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#discussion_r563775391
##########
File path: airflow/migrations/versions/e38be357a868_update_schema_for_smart_sensor.py
##########
@@ -50,6 +51,11 @@ def sa_timestamp(): # noqa: D103
def upgrade(): # noqa: D103
conn = op.get_bind()
+ inspector = Inspector.from_engine(conn)
+ tables = inspector.get_table_names()
+ if "sensor_instance" in tables:
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a change in pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#discussion_r563764874
##########
File path: airflow/migrations/versions/e38be357a868_update_schema_for_smart_sensor.py
##########
@@ -50,6 +51,11 @@ def sa_timestamp(): # noqa: D103
def upgrade(): # noqa: D103
conn = op.get_bind()
+ inspector = Inspector.from_engine(conn)
+ tables = inspector.get_table_names()
+ if "sensor_instance" in tables:
Review comment:
nit: good to use single quotes to keep consistent with the whole script? The same for the change in `downgrade()`
##########
File path: airflow/migrations/versions/e38be357a868_update_schema_for_smart_sensor.py
##########
@@ -50,6 +51,11 @@ def sa_timestamp(): # noqa: D103
def upgrade(): # noqa: D103
conn = op.get_bind()
+ inspector = Inspector.from_engine(conn)
+ tables = inspector.get_table_names()
+ if "sensor_instance" in tables:
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#discussion_r563774796
##########
File path: airflow/migrations/versions/e38be357a868_update_schema_for_smart_sensor.py
##########
@@ -50,6 +51,11 @@ def sa_timestamp(): # noqa: D103
def upgrade(): # noqa: D103
conn = op.get_bind()
+ inspector = Inspector.from_engine(conn)
+ tables = inspector.get_table_names()
+ if "sensor_instance" in tables:
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG merged pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
XD-DENG merged pull request #13892:
URL: https://github.com/apache/airflow/pull/13892
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13892: Make Smart Sensors DB Migration idempotent
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13892:
URL: https://github.com/apache/airflow/pull/13892#issuecomment-766879019
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org