You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ramitkataria (via GitHub)" <gi...@apache.org> on 2025/11/05 00:26:48 UTC

[PR] Fix PK constraint setting + nullable mismatch in callback migration [airflow]

ramitkataria opened a new pull request, #57836:
URL: https://github.com/apache/airflow/pull/57836

   - In some databases like MySQL, setting the primary key constraint while adding the column seems to not work as expected. So, this fixes that by setting the primary key constraint after the column creation statement.
   - There was a mismatch in the nullable property of the `data` column in the callback model. This sets the property to be non-nullable in both the migration script and the model.
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix PK constraint setting + nullable mismatch in callback migration [airflow]

Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on code in PR #57836:
URL: https://github.com/apache/airflow/pull/57836#discussion_r2492810619


##########
airflow-core/src/airflow/migrations/versions/0091_3_2_0_restructure_callback_table.py:
##########
@@ -56,15 +56,16 @@ def upgrade():
     with op.batch_alter_table("callback", schema=None) as batch_op:
         batch_op.add_column(sa.Column("type", sa.String(length=20), nullable=False))
         batch_op.add_column(sa.Column("fetch_method", sa.String(length=20), nullable=False))
-        batch_op.add_column(sa.Column("data", airflow.utils.sqlalchemy.ExtendedJSON(), nullable=True))
+        batch_op.add_column(sa.Column("data", ExtendedJSON(), nullable=False))
         batch_op.add_column(sa.Column("state", sa.String(length=10), nullable=True))
         batch_op.add_column(sa.Column("output", sa.Text(), nullable=True))
         batch_op.add_column(sa.Column("trigger_id", sa.Integer(), nullable=True))
         batch_op.create_foreign_key(batch_op.f("callback_trigger_id_fkey"), "trigger", ["trigger_id"], ["id"])
 
         # Replace INTEGER id with UUID id
         batch_op.drop_column("id")
-        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False, primary_key=True))
+        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False))
+        batch_op.create_primary_key("callback_pkey", ["id"])

Review Comment:
   Thanks for approving! I found the error the original version of this script was causing while running unit tests in the other PR:
   ```
   sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidForeignKey) there is no unique constraint matching given keys for referenced table "callback"
   
   [SQL: ALTER TABLE deadline ADD CONSTRAINT deadline_callback_id_fkey FOREIGN KEY(callback_id) REFERENCES callback (id) ON DELETE CASCADE]
   (Background on this error at: https://sqlalche.me/e/14/f405)
   ```
   This happens with all DB unit tests even in postgres because it doesn't seem to actually create the primary key constraint. This didn't happen in the PR that added this migration script because no other table was referencing the callback table using a foreign key. I wonder why the script didn't raise an error about invalid arg primary_key if it couldn't handle it. But anyway, it should be fixed after this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix PK constraint setting + nullable mismatch in callback migration [airflow]

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck merged PR #57836:
URL: https://github.com/apache/airflow/pull/57836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix PK constraint setting + nullable mismatch in callback migration [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #57836:
URL: https://github.com/apache/airflow/pull/57836#discussion_r2495662487


##########
airflow-core/src/airflow/migrations/versions/0091_3_2_0_restructure_callback_table.py:
##########
@@ -56,15 +56,16 @@ def upgrade():
     with op.batch_alter_table("callback", schema=None) as batch_op:
         batch_op.add_column(sa.Column("type", sa.String(length=20), nullable=False))
         batch_op.add_column(sa.Column("fetch_method", sa.String(length=20), nullable=False))
-        batch_op.add_column(sa.Column("data", airflow.utils.sqlalchemy.ExtendedJSON(), nullable=True))
+        batch_op.add_column(sa.Column("data", ExtendedJSON(), nullable=False))
         batch_op.add_column(sa.Column("state", sa.String(length=10), nullable=True))
         batch_op.add_column(sa.Column("output", sa.Text(), nullable=True))
         batch_op.add_column(sa.Column("trigger_id", sa.Integer(), nullable=True))
         batch_op.create_foreign_key(batch_op.f("callback_trigger_id_fkey"), "trigger", ["trigger_id"], ["id"])
 
         # Replace INTEGER id with UUID id
         batch_op.drop_column("id")
-        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False, primary_key=True))
+        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False))
+        batch_op.create_primary_key("callback_pkey", ["id"])

Review Comment:
   FWIW, I also ran into a similar issue with the migration I am working on at the moment.  The `primary_key-True` value wasn't being honored so I had to use a second line to set it explicitly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix PK constraint setting + nullable mismatch in callback migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57836:
URL: https://github.com/apache/airflow/pull/57836#discussion_r2492589851


##########
airflow-core/src/airflow/migrations/versions/0091_3_2_0_restructure_callback_table.py:
##########
@@ -56,15 +56,16 @@ def upgrade():
     with op.batch_alter_table("callback", schema=None) as batch_op:
         batch_op.add_column(sa.Column("type", sa.String(length=20), nullable=False))
         batch_op.add_column(sa.Column("fetch_method", sa.String(length=20), nullable=False))
-        batch_op.add_column(sa.Column("data", airflow.utils.sqlalchemy.ExtendedJSON(), nullable=True))
+        batch_op.add_column(sa.Column("data", ExtendedJSON(), nullable=False))
         batch_op.add_column(sa.Column("state", sa.String(length=10), nullable=True))
         batch_op.add_column(sa.Column("output", sa.Text(), nullable=True))
         batch_op.add_column(sa.Column("trigger_id", sa.Integer(), nullable=True))
         batch_op.create_foreign_key(batch_op.f("callback_trigger_id_fkey"), "trigger", ["trigger_id"], ["id"])
 
         # Replace INTEGER id with UUID id
         batch_op.drop_column("id")
-        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False, primary_key=True))
+        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False))
+        batch_op.create_primary_key("callback_pkey", ["id"])

Review Comment:
   Is this needed?



##########
airflow-core/src/airflow/migrations/versions/0091_3_2_0_restructure_callback_table.py:
##########
@@ -56,15 +56,16 @@ def upgrade():
     with op.batch_alter_table("callback", schema=None) as batch_op:
         batch_op.add_column(sa.Column("type", sa.String(length=20), nullable=False))
         batch_op.add_column(sa.Column("fetch_method", sa.String(length=20), nullable=False))
-        batch_op.add_column(sa.Column("data", airflow.utils.sqlalchemy.ExtendedJSON(), nullable=True))
+        batch_op.add_column(sa.Column("data", ExtendedJSON(), nullable=False))
         batch_op.add_column(sa.Column("state", sa.String(length=10), nullable=True))
         batch_op.add_column(sa.Column("output", sa.Text(), nullable=True))
         batch_op.add_column(sa.Column("trigger_id", sa.Integer(), nullable=True))
         batch_op.create_foreign_key(batch_op.f("callback_trigger_id_fkey"), "trigger", ["trigger_id"], ["id"])
 
         # Replace INTEGER id with UUID id
         batch_op.drop_column("id")
-        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False, primary_key=True))
+        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False))
+        batch_op.create_primary_key("callback_pkey", ["id"])

Review Comment:
   Is this change needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix PK constraint setting + nullable mismatch in callback migration [airflow]

Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on code in PR #57836:
URL: https://github.com/apache/airflow/pull/57836#discussion_r2492679616


##########
airflow-core/src/airflow/migrations/versions/0091_3_2_0_restructure_callback_table.py:
##########
@@ -56,15 +56,16 @@ def upgrade():
     with op.batch_alter_table("callback", schema=None) as batch_op:
         batch_op.add_column(sa.Column("type", sa.String(length=20), nullable=False))
         batch_op.add_column(sa.Column("fetch_method", sa.String(length=20), nullable=False))
-        batch_op.add_column(sa.Column("data", airflow.utils.sqlalchemy.ExtendedJSON(), nullable=True))
+        batch_op.add_column(sa.Column("data", ExtendedJSON(), nullable=False))
         batch_op.add_column(sa.Column("state", sa.String(length=10), nullable=True))
         batch_op.add_column(sa.Column("output", sa.Text(), nullable=True))
         batch_op.add_column(sa.Column("trigger_id", sa.Integer(), nullable=True))
         batch_op.create_foreign_key(batch_op.f("callback_trigger_id_fkey"), "trigger", ["trigger_id"], ["id"])
 
         # Replace INTEGER id with UUID id
         batch_op.drop_column("id")
-        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False, primary_key=True))
+        batch_op.add_column(sa.Column("id", UUIDType(binary=False), nullable=False))
+        batch_op.create_primary_key("callback_pkey", ["id"])

Review Comment:
   Yes, that helped fix a lot of DB unit tests (MySQL iirc) in #57215 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org