Re: [D] Error observed: AirflowRuntimeError in airflow version 3.1.2 [airflow]

["bkarankar (via GitHub)" <gi...@apache.org>]

GitHub user bkarankar added a comment to the discussion: Error observed: AirflowRuntimeError in airflow version 3.1.2

This looks like a race condition around inserts into rendered_task_instance_fields, where multiple processes attempt to write the same (dag_id, task_id, run_id, map_index) record concurrently. We've seen similar behavior in highly parallel environments (e.g., CeleryExecutor with high worker concurrency).

As a workaround, try disabling rendered task field storage:

AIRFLOW__CORE__STORE_RENDERED_TI_FIELDS=False

Also check your Airflow version, scheduler/API server replica count, and worker concurrency settings. It would be helpful to know whether this is reproducible on the latest Airflow patch release, as there may already be a fix or ongoing work related to this race condition.

GitHub link: https://github.com/apache/airflow/discussions/61442#discussioncomment-17219171

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org