You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/27 15:04:24 UTC
[GitHub] [airflow] nalepae opened a new issue #10604: Upstream failed whereas Upstream is alright
nalepae opened a new issue #10604:
URL: https://github.com/apache/airflow/issues/10604
**Apache Airflow version**: 1.10.3 on Google Cloud Composer (so 1.10.3-composer)
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): (Managed by Google Cloud Composer)
**Environment**:
- **Cloud provider or hardware configuration**: Google Cloud Composer
- **OS** (e.g. from /etc/os-release): (Managed by Google Cloud Composer)
- **Kernel** (e.g. `uname -a`): (Managed by Google Cloud Composer)
- **Install tools**: N/A
- **Others**: Managed by Google Cloud Composer
**What happened**:
Sometimes, some steps do not run because of "Failed Upstream" (orange color) whereas upstream is alright (green color).
_Examples:_
or
_Additional informations:_
- All steps except `wait_for_dump`, `recort_start`, `join` and `record_end` are Kubernetes Pod Operator.
- Re running by hand "orange" steps always solve the issue.
**What you expected to happen**:
If all upstream tasks of a given task are green, the given task should run, and not be in state "Failed Uptream".
**How to reproduce it**:
Run some DAGs (no need to be a specific DAG) on Airflow (via Google Cloud Composer), and the issue will happen randomly.
**Anything else we need to know**:
I cannot reproduce this bug on my local version on Airflow, on MacOS (after about 10 trials) using the **Kubernetes single-node cluster** available in **Docker for mac**.
I'm not 100% sure this issue come from **Airflow** itself, but may also come from **Google Cloud Composer**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dlachasse commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
dlachasse commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-953138832
We're experiencing this as well on Airflow 2.1.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nalepae commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
nalepae commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-700965664
@adambayrami we still have the issue. What about you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubhamg931 commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
shubhamg931 commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-931249828
facing this again with airflow 2.1.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nalepae commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
nalepae commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-719474023
Fixed when upgrading to **Airflow 1.10.10**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-953592699
Please open a new issue instead of replying to an old, fixed one. Please also provide more information about your environment when you do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] adambayrami commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
adambayrami commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-698379771
Seeing the same issue on airflow 1.10.5. We recently increased from about 30 dags and 200-300 tasks to about 80 dags and over 1k task pretty quickly then started seeing this issue.
Using reddis with a celery executor. Don't currently have access to scheduler logs but that's where I'd like to start investigating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] LeandroDoldan commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
LeandroDoldan commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-968843610
Facing it as well on Airflow 2.1.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] adambayrami commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
adambayrami commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-698379771
Seeing the same issue on airflow 1.10.5. We recently increased from about 30 dags and 200-300 tasks to about 80 dags and over 1k task pretty quickly then started seeing this issue.
Using reddis with a celery executor. Don't currently have access to scheduler logs but that's where I'd like to start investigating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] adambayrami commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
adambayrami commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-700964334
Did you all learn anything new or do you still have the issue? @pquentin @nalepae
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pquentin commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
pquentin commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-698759394
I work with @nalepae, and we continue to see this a few times per week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pquentin commented on issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
pquentin commented on issue #10604:
URL: https://github.com/apache/airflow/issues/10604#issuecomment-698759394
I work with @nalepae, and we continue to see this a few times per week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nalepae closed issue #10604: Upstream failed whereas Upstream is alright
Posted by GitBox <gi...@apache.org>.
nalepae closed issue #10604:
URL: https://github.com/apache/airflow/issues/10604
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org