You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/04 21:36:11 UTC
[GitHub] [airflow] vxtals opened a new issue #12823: DAG callback handling not working properly on simultaneous executions
vxtals opened a new issue #12823:
URL: https://github.com/apache/airflow/issues/12823
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**:
apache-airflow 2.0.0b3
**Environment**:
Any environment
**What happened**:
I have dags triggered at same time for several other Dags through TriggerDagRunOperator (let's say this dags have a minute schedule). Then, on_success_callback is not triggered for some of this dags.
**What you expected to happen**:
Airflow is working pretty fast and sometimes executions_date for these dags is in the same second, so the execution_date stored in the DB is the same, when it calls _execute_dag_callbacks in scheduler_job.py and retrieve DagRun using this date the dag is not the right one.
**How to reproduce it**:
Create a dag with any task triggered by two other dags with the same schedule, every minute or every 5 minutes will be the faster way of testing.
I'm attaching a file with the 3 dags on it as txt
[multi_trigger.txt](https://github.com/apache/airflow/files/5645616/multi_trigger.txt)
, starting them and waiting to any two consecutive executions with the same execution_date and checking scheduler logs should be enough to reproduce it.
**Anything else we need to know**:
It doesn't happen in every execution but happens often, and I guess it's more likely with a better machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vxtals commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
vxtals commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-780545854
I'm gonna ping this issue...
Could someone confirm if I'm right about this weird behavior? I'm experiencing other problems related to this, for example, macro {{ dag_run.id }} will have the same value for two runs at the same time. Going crazy with this...
Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-780637688
Does this happen on Airflow 2.0.1 too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cvsekhar commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
cvsekhar commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-847135638
This issue is happening in the version
Version: v2.0.0
Git Version:.release:2.0.0+ab5f770bfcd8c690cbe4d0825896325aca0beeca also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cvsekhar commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
cvsekhar commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-866486350
Hi, that works perfect with azurefileshare thank you for the quick fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cvsekhar commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
cvsekhar commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-847924506
@eladkal sure, will post update after I upgrade to 2.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-847922219
@cvsekhar can you check if it happens in 2.1.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cvsekhar edited a comment on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
cvsekhar edited a comment on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-866486350
Hi @kaxil , that works perfect with azurefileshare thank you for the quick fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-865927723
Any updates @cvsekhar ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #12823:
URL: https://github.com/apache/airflow/issues/12823
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #12823: DAG callback handling not working properly on simultaneous executions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12823:
URL: https://github.com/apache/airflow/issues/12823#issuecomment-831609016
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org