You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/27 09:00:29 UTC
[GitHub] [airflow] sabarisan11 opened a new issue #17254: Airflow dependencies not working in 1.10.15
sabarisan11 opened a new issue #17254:
URL: https://github.com/apache/airflow/issues/17254
Hi Everyone,
We are trying to migrate our code from Airflow version **1.10.5** to **1.10.15** (bridge version) and having plans to use Airflow 2.0. But post migrating our code into 1.10.15 we are facing some dependency issues where the same code works fine in 1.10.5.
When looked further, we identified that upstream and downstream functions with parameter is causing the issue. we tried modifying it with >> & << and that works. We have around 100 DAG in our system and it would be a huge task to modify all 100 DAG and test those. we also have some interdependencies like Task A is upstream to 3 tasks & also serves as downstream to 5 other tasks. In such cases it would be really difficult to go for some automated way to modify our code
**Airflow Hanging Dependencies in 1.10.15**
In graph view, dependencies appear like they are not properly connected but are actually connected. If I check the task instance details in this, upstream and downstream are set properly but it doesn't appear as expected
**Same Code works in Airflow 1.10.5**
where the same code works properly and dependencies shows up in 1.10.5 version
**Sample Code snippet**
Code block that has function parameters, here if the dependencies is between the operator it works fine, but when we brought python function which returns operator that is working well for past 2 years in Airflow version 1.10.5 but shows up this graph dependency issues in new version
Kindly take a look at this and provide some advice on how to proceed further
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17254: Airflow dependencies not working in 1.10.15
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17254:
URL: https://github.com/apache/airflow/issues/17254#issuecomment-887339206
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17254: Airflow dependencies not working in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17254:
URL: https://github.com/apache/airflow/issues/17254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17254: Airflow dependencies not working in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17254:
URL: https://github.com/apache/airflow/issues/17254#issuecomment-887378263
The 1.10 version of Airflow reached and of life as of June 17 2021 and it will not receive any more fixes (even critical ones). This looks like some If the code does not work in 1.10.15, it is highly unlikely it is going to work in Airflow 2 (which we encourage everyone to use). I am afraid fixing your DAGs to work with 1.10.15 is the only way to proceed. Since you already know a solution that works, I think this is the only way you can proceed.
Since we are not going to do anything with the issue, I am closing this one - feel free to copy it to a Github discussion instead if you have hopes that you will find someone with similar issue and a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17254: Airflow dependencies not working in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17254:
URL: https://github.com/apache/airflow/issues/17254#issuecomment-887378263
The 1.10 version of Airflow reached and of life as of June 17 2021 and it will not receive any more fixes (even critical ones). This looks like some of the code of yours does not work in 1.10.15 and it was a buggy behaviour in 1.10.5. It is highly unlikely it is going to work in Airflow 2 (which we encourage everyone to use). I am afraid fixing your DAGs to work with 1.10.15 is the only way to proceed. Since you already know a solution that works, I think this is the only way you can proceed.
Since we are not going to do anything with the issue, I am closing this one - feel free to copy it to a Github discussion instead if you have hopes that you will find someone with similar issue and a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17254: Airflow dependencies not working in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17254:
URL: https://github.com/apache/airflow/issues/17254#issuecomment-887378263
The 1.10 version of Airflow reached and of life as of June 17 2020 and it will not receive any more fixes (even critical ones). This looks like some If the code does not work in 1.10.15, it is highly unlikely it is going to work in Airflow 2 (which we encourage everyone to use). I am afraid fixing your DAGs to work with 1.10.15 is the only way to proceed. Since you already know it works, I think this is the only way you can proceed.
Since we are not going to do anything with the issue, I am closing this one - feel free to copy it to a Github discussion instead if you have hopes that you will find someone with similar issue and a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17254: Airflow dependencies not working in 1.10.15
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17254:
URL: https://github.com/apache/airflow/issues/17254#issuecomment-887378263
The 1.10 version of Airflow reached and of life as of June 17 2020 and it will not receive any more fixes (even critical ones). This looks like some If the code does not work in 1.10.15, it is highly unlikely it is going to work in Airflow 2 (which we encourage everyone to use). I am afraid fixing your DAGs to work with 1.10.15 is the only way to proceed. Since you already know a solution that works, I think this is the only way you can proceed.
Since we are not going to do anything with the issue, I am closing this one - feel free to copy it to a Github discussion instead if you have hopes that you will find someone with similar issue and a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org