You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/08 16:03:26 UTC
[GitHub] [airflow] tooptoop4 opened a new issue #11354: "trying to overwrite a task will raise an exception"
tooptoop4 opened a new issue #11354:
URL: https://github.com/apache/airflow/issues/11354
v1.10.9 was fine
i upgraded to v 1.10.12 and started getting below error after running get_state or clear on the cli:
/home/ec2-user/venv/lib/python3.7/site-packages/airflow/models/dag.py:1350: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id COMPLETE is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)
i am using the same taskid (named COMPLETE) on 2 different dags, in fact i receive this warning when running the get_state cli on a different (ie 3rd) dag (a that does not contain task_id COMPLETE)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-705697414
similar to https://bigdata-etl.com/apache-airflow-create-dynamic-dag/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-857273138
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Alibm73 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
Alibm73 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-707578501
@kaxil @tooptoop4
So my solution was setting ```load_examples = False``` in the ```airflow.cfg```. Now the webserver started without any error.
I don't know this helps or not but figured posting it doesn't hurt #8728
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-705669561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-773307556
@kaxil can this be reopened? https://github.com/apache/airflow/commit/39c91d985b8249b78893e3d0d8145a9a2693340e#diff-16df48936f837b402448fbbff41aaba51279da9c9b66d4b39412ab9d8237eedd does not solve it but reverting https://github.com/apache/airflow/pull/8728 does
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-773319134
reverting #8728 will hide the problem and users will face issues when they upgrade to 2.0.0
Can you please provide me an example dag/dag file where I can reproduce (not link to any blog post)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-705669561
Can you post your DAG for us to reproduce please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-711076551
I already had load_examples = False but same issue. Reverting https://github.com/apache/airflow/pull/8728 fixes for me
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-705682957
can't share but they are dynamic dag generators (a single .py creates 5 different dagids, then another .py creates 3 different dagids) , makes use of globals()[dag_id].
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-773307556
@kaxil can this be reopened? https://github.com/apache/airflow/commit/39c91d985b8249b78893e3d0d8145a9a2693340e#diff-16df48936f837b402448fbbff41aaba51279da9c9b66d4b39412ab9d8237eedd does not solve it but reverting https://github.com/apache/airflow/pull/8728 does
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] epx-dan commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
epx-dan commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-731304973
I have the same issue after airflow webserver:
`PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)`
I turned load_exampes from True to False in airflow.cfg, then this warning disappeared.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-705682957
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-773319134
reverting #8728 will hide the problem and users will face issues when they upgrade to 2.0.0
Can you please provide me an example dag/dag file where I can reproduce (not link to any blog post)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #11354:
URL: https://github.com/apache/airflow/issues/11354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Alibm73 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
Alibm73 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-706971626
@kaxil @tooptoop4 have the same issue.
```sh
airflow version
1.10.12
```
when i type ```airflow webserver -p 7000``` i get this error
```
airflow webserver -p 7000
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-10-12 12:00:23,636] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:23,637] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:7000
Timeout: 120
Logfiles: - -
=================================================================
[2020-10-12 12:00:26 +0330] [3064] [INFO] Starting gunicorn 20.0.4
[2020-10-12 12:00:26 +0330] [3064] [INFO] Listening at: http://0.0.0.0:7000 (3064)
[2020-10-12 12:00:26 +0330] [3064] [INFO] Using worker: sync
[2020-10-12 12:00:26 +0330] [3100] [INFO] Booting worker with pid: 3100
[2020-10-12 12:00:26 +0330] [3101] [INFO] Booting worker with pid: 3101
[2020-10-12 12:00:26 +0330] [3103] [INFO] Booting worker with pid: 3103
[2020-10-12 12:00:26 +0330] [3107] [INFO] Booting worker with pid: 3107
[2020-10-12 12:00:27,792] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,793] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
[2020-10-12 12:00:27,807] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,809] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
[2020-10-12 12:00:27,818] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,820] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
[2020-10-12 12:00:27,819] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,820] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)
^C[2020-10-12 12:00:52 +0330] [3064] [INFO] Handling signal: int
[2020-10-12 12:00:52,299] {cli.py:1127} INFO - Received signal: 2. Closing gunicorn.
[2020-10-12 12:00:52 +0330] [3101] [INFO] Worker exiting (pid: 3101)
[2020-10-12 12:00:52 +0330] [3100] [INFO] Worker exiting (pid: 3100)
[2020-10-12 12:00:52 +0330] [3103] [INFO] Worker exiting (pid: 3103)
[2020-10-12 12:00:52 +0330] [3107] [INFO] Worker exiting (pid: 3107)
[2020-10-12 12:00:53 +0330] [3064] [INFO] Shutting down: Master
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-705689586
I mean can you create a dummy DAG that you can share
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-731308717
> I have the same issue after airflow webserver:
> `PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception. category=PendingDeprecationWarning)`
> I turned load_exampes from True to False in airflow.cfg, then this warning disappeared.
This is fixed in 1.10.13
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 commented on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-707257224
local path on my 1.12 to revert https://github.com/apache/airflow/pull/8728 fixes this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tooptoop4 edited a comment on issue #11354: "trying to overwrite a task will raise an exception"
Posted by GitBox <gi...@apache.org>.
tooptoop4 edited a comment on issue #11354:
URL: https://github.com/apache/airflow/issues/11354#issuecomment-707257224
local patch on my 1.12 to revert https://github.com/apache/airflow/pull/8728 fixes this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org