[PR] Cloud SQL – Handle 409 OPERATION_IN_PROGRESS for concurrent admin operations [airflow]

["shashbha14 (via GitHub)" <gi...@apache.org>]

shashbha14 opened a new pull request, #62674:
URL: https://github.com/apache/airflow/pull/62674

   Summary
   This patch for the `google.cloud.sql` module for Airflow allows for only one administrative operation at a time for an instance (import, export, clone, patch, etc.). When multiple operations are executed concurrently, the API returns HTTP 409 with the reason `operation_in_progress`. This patch includes an orchestration-level sensor as well as an operator-level retry, so that the problem is addressed by the DAGs themselves, avoiding the need for workarounds.
   
   Fixes #62671
   
    Changes
   
   ### Option A: Deferrable sensor (orchestration)
   - **CloudSQLInstanceOperationsSensor**: This sensor will wait until there is no PENDING/RUNNING operation running on the instance. This is instance-aware, so it will work across multiple DAGs.
   - **CloudSQLInstanceOperationsTrigger**: This is an async trigger, which is used by the sensor.
   - **list_instance_operations**: This is a new method for `CloudSQLHook` as well as `CloudSQLAsyncHook`, which is used to list the operations for an instance, using the Operations API.
   
   ### Option B: Operator-level retry
   - **@operation_in_progress_retry()** is applied to `import_instance`, `export_instance`, `clone_instance` methods of `CloudSQLHook`. This will retry the call with exponential backoff when the HTTP 409 status is returned with the status `operation_in_progress`, as is done with `GoogleBaseHook`.
   
   ### Other
   - The sensor is added to the provider info, as well as unit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Cloud SQL – Handle 409 OPERATION_IN_PROGRESS for concurrent admin operations [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal commented on PR #62674:
URL: https://github.com/apache/airflow/pull/62674#issuecomment-3986127374

   > Why are changes for partitioning DAG Runs in this PR?
   
   Because it's AI spam.
   Not worth our timem closing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Cloud SQL – Handle 409 OPERATION_IN_PROGRESS for concurrent admin operations [airflow]

["jroachgolf84 (via GitHub)" <gi...@apache.org>]

jroachgolf84 commented on PR #62674:
URL: https://github.com/apache/airflow/pull/62674#issuecomment-3986084678

   Why are changes for partitioning DAG Runs in this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Cloud SQL – Handle 409 OPERATION_IN_PROGRESS for concurrent admin operations [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal closed pull request #62674: Cloud SQL – Handle 409 OPERATION_IN_PROGRESS for concurrent admin operations
URL: https://github.com/apache/airflow/pull/62674


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org