You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "SameerMesiah97 (via GitHub)" <gi...@apache.org> on 2026/02/13 01:24:35 UTC

[I] ComputeEngineInsertInstanceOperator ignores configuration changes when instance already exists [airflow]

SameerMesiah97 opened a new issue, #61829:
URL: https://github.com/apache/airflow/issues/61829

### Apache Airflow Provider(s)

google

### Versions of Apache Airflow Providers

`apache-airflow-providers-google==20.0.0rc1`

### Apache Airflow version

main

### Operating System

Debian GNU/Linux 12 (bookworm)

### Deployment

Other

### Deployment details

_No response_

### What happened

`ComputeEngineInsertInstanceOperator` currently treats the mere presence of a Compute Engine instance as success. When an instance already exists, the operator logs that it exists and immediately returns success without verifying whether the existing resource matches the requested configuration. As a result, configuration differences are not detected on subsequent DAG runs—particularly changes to critical fields such as `machine_type` or boot disk settings. If these values are modified in the DAG and the task is re-run, the operator neither recreates nor updates the instance; it simply succeeds silently. This leads to incorrect idempotence semantics and allows infrastructure drift to persist undetected across DAG executions.

### What you think should happen instead

The operator should verify that the existing Compute Engine instance matches the configuration defined in the DAG rather than relying solely on presence-based idempotence. On subsequent DAG runs, it should detect differences in critical configuration fields—such as `machine_type` and disk parameters—and surface those differences explicitly. At minimum, configuration drift should be logged so that users are aware of the mismatch. Ideally, the operator should support reconciling the resource to the declared state, for example by recreating the instance when differences are detected and an explicit flag is set. This would ensure consistent, declarative behavior across DAG re-runs and prevent silent infrastructure drift.

### How to reproduce

1. Configure a Google Cloud connection in Airflow (e.g. `google_cloud_default`) with a service account that has permission to create and manage Compute Engine instances.
2. Ensure you have a valid GCP project ID and zone (for example, `us-central1-a`).
3. Create the following DAG:
```
from airflow import DAG
from datetime import datetime
from airflow.providers.google.cloud.operators.compute import (
ComputeEngineInsertInstanceOperator,
)

PROJECT_ID = "<YOUR_PROJECT_ID>"
ZONE = "us-central1-a"
INSTANCE_NAME = "airflow-idempotence-test"

with DAG(
dag_id="gce_idempotence_repro",
start_date=datetime(2025, 1, 1),
schedule=None,
catchup=False,
) as dag:

create_instance = ComputeEngineInsertInstanceOperator(
task_id="create_instance",
project_id=PROJECT_ID,
zone=ZONE,
body={
"name": INSTANCE_NAME,
"machine_type": f"zones/{ZONE}/machineTypes/e2-medium", # Initial machine type used in this repro
"disks": [
{
"boot": True,
"auto_delete": True,
"initialize_params": {
"source_image": "projects/debian-cloud/global/images/family/debian-11"
},
}
],
"network_interfaces": [
{
"network": "global/networks/default"
}
],
},
)
```
4. Trigger the DAG once and confirm that the instance is created successfully.
5. Update the machine type:

`"machine_type": f"zones/{ZONE}/machineTypes/n2-standard-4",`
(Any valid machine type different from the original may be used; in this repro, `n2-standard-4` is used.)

6. Trigger the DAG again.

**Observed Behavior**

The task logs that the instance already exists and completes successfully. However, the instance configuration remains unchanged (e.g., the machine type stays `e2-medium`), and no configuration differences are detected or logged. This shows that changes to fields such as `machine_type` (and disk configuration) are not recognized on DAG re-run, and the operator does not reconcile the resource to the requested state.

### Anything else

**This report is not proposing destructive behavior by default**. Automatically deleting and recreating instances when differences are detected may not be appropriate for all users or environments. However, at a minimum, configuration differences should be surfaced in logs. Any reconciliation behavior (such as recreation) should be explicitly opt-in, allowing users to choose stronger convergence semantics when desired without changing existing default behavior.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ComputeEngineInsertInstanceOperator ignores configuration changes when instance already exists [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.

shahar1 closed issue #61829: ComputeEngineInsertInstanceOperator ignores configuration changes when instance already exists
URL: https://github.com/apache/airflow/issues/61829


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ComputeEngineInsertInstanceOperator ignores configuration changes when instance already exists [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on issue #61829:
URL: https://github.com/apache/airflow/issues/61829#issuecomment-3986402179

   @SameerMesiah97 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
   
   This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
   
   We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ComputeEngineInsertInstanceOperator ignores configuration changes when instance already exists [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.

SameerMesiah97 commented on issue #61829:
URL: https://github.com/apache/airflow/issues/61829#issuecomment-3986735894

   > [@SameerMesiah97](https://github.com/SameerMesiah97) We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
   > 
   > This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
   > 
   > We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
   
   Already has an open PR. Is this an automated message?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org