You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/20 06:59:22 UTC

[GitHub] [airflow] NonlinearNimesh opened a new issue, #28486: DAG Run conflict with dag_id and logical date

NonlinearNimesh opened a new issue, #28486:
URL: https://github.com/apache/airflow/issues/28486

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   I am triggering an airflow dag with an api but it is throwing me this response
   ```
   {
     "detail": "DAGRun with DAG ID: 'working_1' and DAGRun logical date: '2022-11-03 06:35:14+00:00' already exists",
     "status": 409,
     "title": "Conflict",
     "type": "https://airflow.apache.org/docs/apache-airflow/2.3.3/stable-rest-api-ref.html#section/Errors/AlreadyExists"
   }
   ```
   and this is becasue i am triggering multiple dag runs in 1 seconds with same DAG ID. Below is my code
   
   ```
   utc_date = datetime.datetime.utcnow()
   formatted_date = utc_date.strftime('%Y-%m-%dT%H:%M:%SZ')
   
   data = {
       "conf": {},
       "replace_microseconds":"false"
       "dag_run_id": str(uuid.uuid4()),
       "logical_date": str(formatted_date),
   }
   headers={
       'Content-type':'application/json',
       'Accept':'application/json'
   }
   json_payload = json.dumps(data)
   r = requests.post("http://localhost:8080/api/v1/dags/working_1/dagRuns", auth=HTTPBasicAuth("airflow", "airflow"), data=json_payload, headers=headers)
   print(r.status_code)
   print(r.text)
   ```
   
   ### What you think should happen instead
   
   In "https://github.com/apache/airflow/pull/15174"  @potiuk  mentioned that **"In Airflow 2.2 the primary key is run_id and execution_date (Now logical_date)"** and as you can see in the code dag_run_id is always unique so even if logical date is same between two api request then also there will be any conflict. yes i am getting the conflict error .
   
   Like i want to hit airflow api multiple time in a single seconds and according to my understanding of what @potiuk mentioned in the above mentioned link with (dag_run_id + logical_date) will always be unique.
   
   ### How to reproduce
   
   You can use the above code and hit api multiple times (For example 20-30 times) then you will get this conflict error.
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   2.3.3
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #28486: DAG Run conflict with dag_id and logical date

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28486:
URL: https://github.com/apache/airflow/issues/28486#issuecomment-1358917319

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr closed issue #28486: DAG Run conflict with dag_id and logical date

Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #28486: DAG Run conflict with dag_id and logical date
URL: https://github.com/apache/airflow/issues/28486


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on issue #28486: DAG Run conflict with dag_id and logical date

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #28486:
URL: https://github.com/apache/airflow/issues/28486#issuecomment-1358964733

   Unfortunately there are still too many things in the ecosystem (including some parts of Airflow itself) that depends on the logical/execution date being unique that it is unlikely for us to lift this restriction. I’m going to close this since there is no clear goal we can work toward, but feel free to open discussions (in GitHub Discussions or on the dev mailing list) to propose a more concrete plan to eventually migrate away from this constraint.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org