You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/20 16:46:51 UTC

[GitHub] [airflow] sdebruyn opened a new issue, #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

sdebruyn opened a new issue, #28499:
URL: https://github.com/apache/airflow/issues/28499

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   Airflow 2.4.3
   
   Just like in issue #26599 I get a lot of warnings saying I'm registering dependencies multiple times. With a little bit of debugging, I found the culprit:.
   
   Some of my tasks have multiple outputs, which are then used by other tasks. These are TaskFlow-based tasks. The task dependency is added when it detects the first XComArg and then it's added again with every following XComArg, generating these warnings.
   
   Call stack:
   
   <img width="319" alt="image" src="https://user-images.githubusercontent.com/963413/208720058-bf66a57c-8849-4fda-9bd4-b570eac4ce93.png">
   
   
   ### What you think should happen instead
   
   You should not see the warnings if the cause is that you're using multiple outputs from the same upstream task.
   
   ### How to reproduce
   
   Create a function that returns a `dict[str, str]` and make it a task with `@task(multiple_outputs=True)`. Then create another task that takes multiple of these outputs as input.
   
   ### Operating System
   
   Debian GNU/Linux 11 (bullseye)
   
   ### Versions of Apache Airflow Providers
   
   not relevant
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   You can get this with unit tests just by creating a `DagBag` instance with your DAGs.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy closed issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy closed issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs
URL: https://github.com/apache/airflow/issues/28499


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1620244385

   Fixed in https://github.com/apache/airflow/pull/31502


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vedantlodha commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "vedantlodha (via GitHub)" <gi...@apache.org>.
vedantlodha commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1494063197

   @pierrejeambrun @potiuk I dont see any progreess on this issue for a while. Can i take this up if no one else is working? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ephraimbuddy commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1366545805

   I think we should do away with this warning


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] justusel commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "justusel (via GitHub)" <gi...@apache.org>.
justusel commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1470303316

   Hi,
   we are currently facing the same issue with Airflow v2.5.1. We are collecting almost 100mb scheduler logs each day. Sorting out these log files is a temporary solution, but also it's not possible to easily detect independent errors/warnings in the log files that way. Is a fix planned for the next release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] andrew-stein-sp commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "andrew-stein-sp (via GitHub)" <gi...@apache.org>.
andrew-stein-sp commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1545900624

   Just discovered this morning that we're ingesting 2.3 BILLION of these types of logs from airflow into elasticsearch every month... please do away with this warning.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] arturbrandysonwelo commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "arturbrandysonwelo (via GitHub)" <gi...@apache.org>.
arturbrandysonwelo commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1549665848

   As a quick workaround, you can set the logging level from INFO to WARNING in _airflow.cfg_ file:
   ```
   [logging]
   
   logging_level = WARNING
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] arturbrandysonwelo commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "arturbrandysonwelo (via GitHub)" <gi...@apache.org>.
arturbrandysonwelo commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1549406891

   We have this same problem on version 2.5.3 every of our DAGs is disabled right now (because we are moving from an old production environment to a new one) but airflow logs are growing very fast in the log partition.  They are full of Warnings - Dependecy task already registered for DAG.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ketozhang commented on issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

Posted by "ketozhang (via GitHub)" <gi...@apache.org>.
ketozhang commented on issue #28499:
URL: https://github.com/apache/airflow/issues/28499#issuecomment-1583658710

   Although turning off the warning is pretty easy, there seems to be an underlying inefficiency of re-registering tasks when using Taskflow with multiple output, dynamic generation of tasks, and dynamic task mapping.
   
   @vedantlodha Any thoughts from what you've worked on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org