You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/11 15:20:16 UTC

[GitHub] [airflow] MaximM95 opened a new issue, #25679: Dag that runs a GcsSensorOperator creates always 1 running instance and queues 16 in parallel when deploying the dag code.

MaximM95 opened a new issue, #25679:
URL: https://github.com/apache/airflow/issues/25679

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google==5.1.0
   
   ### Apache Airflow version
   
   2.1.4
   
   ### Operating System
   
   Google Cloud Composer (GKE Autopilot, composer-2.0.22-airflow-2.1.4)
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   Deployment via Terraform, see https://cloud.google.com/composer/docs/composer-2/create-environments#step_optional_select_a_service_account_for_your_environment
   
   ### What happened
   
   We recently upgraded our Cloud Composer from version Composer-1.17.2-Airflow-2.1.2 to Composer-2.0.22-Airflow-2.1.4
   
   When deploying a dag which runs a custom GcsBucketSensor that inherits from BaseSensorOperator, one instance runs and 16 others are queued. If I close one, another one is queued or If one sensor task has finished successful than another one will run and one further will be queued. It means that the scheduler is permanently trying to run 17 instances in parallel. 
   
   That is the dag config.
   dag = DAG(
        dag_id=Path(__file__).stem,
        schedule_interval=timedelta(seconds=1),
        start_date=days_ago(1),
        default_args={},
        max_active_runs=1,
        catchup=False
   
   What could be the reason that the Composer Scheduler scheduled 17 DagRuns?
   
   
   
   
   
   ### What you think should happen instead
   
   Only 1 active DagRun without permanently queueing 16 further DagRuns.
   
   ### How to reproduce
   
   1. Create a Composer-2.0.22-Airflow-2.1.4 environment on GCP
   2. Create a DAG that runs a sensor that inherits from BaseSensorOperator
   3. Deploy/Upload the dag code to the /dags folder and the sensor to the /plugins folder in the Composer Bucket
   4. The sensor will run automatically
   
   
   
   ### Anything else
   
   In Composer v1 this problem has never occured. This problem does not destroy anything in the dag but it looks weird in the logs that always 16 further dags are scheduled.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #25679: Dag that runs a GcsSensorOperator creates always 1 running instance and queues 16 in parallel when deploying the dag code.

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #25679: Dag that runs a GcsSensorOperator creates always 1 running instance and queues 16 in parallel when deploying the dag code.
URL: https://github.com/apache/airflow/issues/25679


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #25679: Dag that runs a GcsSensorOperator creates always 1 running instance and queues 16 in parallel when deploying the dag code.

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25679:
URL: https://github.com/apache/airflow/issues/25679#issuecomment-1212136720

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org