[GitHub] [airflow] khyurri commented on a change in pull request #10254: Warning about unsafe migrations

[GitBox <gi...@apache.org>]

khyurri commented on a change in pull request #10254:
URL: https://github.com/apache/airflow/pull/10254#discussion_r499124977



##########
File path: airflow/utils/db.py
##########
@@ -606,6 +606,59 @@ def check_migrations(timeout):
             log.info('Waiting for migrations... %s second(s)', ticker)
 
 
+def check_conn_id_duplicates(session=None):
+    """
+    Check unique conn_id in connection table
+    @param session:  session of the sqlalchemy
+    @return: str
+    """
+    dups = session.query(Connection, func.count(Connection.conn_id)) \
+                  .group_by(Connection.conn_id) \
+                  .having(func.count(Connection.conn_id) > 1).all()
+    if dups:
+        return f'Seems you have non unique conn_id in connection table.\n' \
+               f'You have to manage those duplicate connections ' \
+               f'before upgrading the database.\n' \
+               f'Duplicated conn_id: {[dup[0] for dup in dups]}'
+    return ''
+
+
+def check_conn_type_null(session=None):
+    """
+    Check nullable conn_type column in connection table
+    @param session:  session of the sqlalchemy
+    @return: str
+    """
+    n_nulls = session.query(Connection)\
+                     .filter(Connection.conn_type.is_(None)).all()
+    if n_nulls:
+        return f'The conn_type column in the connection ' \
+               f'table must contain content.\n' \
+               f'Make sure you don\'t have null ' \
+               f'in the conn_type column.\n' \
+               f'Null conn_type conn_id: {list(n_nulls)}'
+    return ''
+
+
+@provide_session
+def auto_migrations_available(session=None):
+    """
+    @session: session of the sqlalchemy
+    @return: list[str]
+    """
+    errors_ = []
+    try:
+        for check_fn in (check_conn_id_duplicates, check_conn_type_null):
+            err = check_fn(session)
+            if err:
+                errors_.append(err)
+    except (exc.OperationalError, exc.ProgrammingError):
+        # fallback if tables hasn't been created yet
+        pass

Review comment:
       >  I would expect that if Connection table does not exits, other tables won't exist either.
   
   Agree. 
   If `exc.OperationalError, exc.ProgrammingError`, it means, that database in unexpected state (eg empty). No reason to continue check migration safety. 
   
   Other way: 
   The script can except `exc.OperationalError, exc.ProgrammingError` on every item in loop:
   
   ```python
   for check_fn in (check_conn_id_duplicates, check_conn_type_null):
       try:
           err = check_fn(session)
           if err:
               errors_.append(err)
       except (exc.OperationalError, exc.ProgrammingError):
            # fallback if table hasn't been created yet
           pass
   ```
   
   What do you think about it? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org