You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/31 08:08:27 UTC
[GitHub] [airflow] potiuk opened a new issue, #24037: Frequent failures of helm chart tests
potiuk opened a new issue, #24037:
URL: https://github.com/apache/airflow/issues/24037
### Apache Airflow version
main (development)
### What happened
We keep on getting very frequent failures of Helm Chart tests and seems that a big number of those errors are because of errors when pulling charts from bitnami for postgres:
Example here (but I saw it happening very often recently):
https://github.com/apache/airflow/runs/6666449965?check_suite_focus=true#step:9:314
```
Save error occurred: could not find : chart postgresql not found in https://charts.bitnami.com/bitnami: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR
Deleting newly downloaded charts, restoring pre-update state
Error: could not find : chart postgresql not found in https://charts.bitnami.com/bitnami: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR
Dumping logs from KinD
```
It is not only a problem for our CI but it might be similar problem for our users who want to install the chart - they might also get the same kinds of error.
I guess we should either make it more resilient to intermittent problems with bitnami charts or use another chart (or maybe even host the chart ourselves somewhere within apache infrastructure. While the postgres chart is not really needed for most "production" users, it is still a dependency of our chart and it makes our chart depend on external and apparently flaky service.
### What you think should happen instead
We should find (or host ourselves) more stable dependency or get rid of it.
### How to reproduce
Look at some recent CI builds and see that they often fail in K8S tests and more often than not the reason is missing postgresql chart.
### Operating System
any
### Versions of Apache Airflow Providers
not relevant
### Deployment
Other
### Deployment details
CI
### Anything else
Happy to make the change once we agree what's the best way :).
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1143794823
We can also vendor it in chart/charts. Have we tried to reach out to the bitnami folks to see if they are aware of the issues and working to resolve them though? That's the easiest for us, at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1145186532
Yeah I think so. Happy to do it to learn a bit more. That's great it's vendored in in the tarball.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1144909620
So taking itnt account what happened here tonight https://github.com/bitnami/charts/issues/10539 - I believe vendoring-in the chart seems to be the best idea.
I REALLY don't like when someone arbitrary decision can break all our released versions of charts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1144125178
I've opened #24089 to point at github for the index instead. This will stop the bleeding at least and give us a little time to see if they resolve it (sounds like it is now causing them pain, so I'm hopeful) or if we need to do something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1143991231
Someone opened an issue to bitnami charts about it iliterally 15 minutes ago. Commented it there: https://github.com/bitnami/charts/issues/10535#issuecomment-1143990338
Any upvotes, cheering comments are most welcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1144003498
Speaking of "supporting free use" - they just closed the issue and redirected it to issue they are "working on" https://github.com/bitnami/charts/issues/8433 which has been open 16 Dec 2021.
@jedcunningham - do you still think it's the "easiest route" :P ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1144005310
BTW. Rough looking at the issue about half of the ~50 comments there is "we have the same issue - here are the details" and the other half (exaggerating a bit for the effect of course) is "Thanks, we are working on it" from bitnami support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1141808094
cc: @kaxil @jedcunningham @ephraimbuddy - I think it would be great to find a better way. We do not really need that postgres chart to be frequently updated, so hosting it ourselves (say in a github repo) is I think the best option.
WDYT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1143980447
Surely - we can ask, but they are part of vmware so I have no high expectations their free offering will be looked at carefully. But I definitely will.
However I think we should seriously consider moving it to Github (or vendoring in) regardless of their answers. What do we do if they answer "we fixed it" ? It happens intermittently so we are not even able to check if they did.
I really do not like situations where WE have to carry the burden of CI errors (and make our contributors unhappy) when someone else screwed up.
I am not sure if this is a serious effort - this is one-time effort really which needs zero maintenance (maybe upgrade from time to time). Comparing with multiple unforeseen errors that are yet another reson for our users to learn that "red" is normal and reaching out for help. I think the key to keep CI "nice" for the users is to eliminate relentlessly any reasons for potential problems - leaving only those that are "real" problems as false negatives undermine your trust in it and the more it happens the more you reach out to maintainers "the tests failed, please help" - we should keep the possibility down to minimum. It can never be eliminated entirely but if we do not have control over something we cannot improve it.
I've heard very similar concerns when I moved the images used during our integration tests.
And this worked beautifully after I did this for all the images we use in our integration tests in ghcr.io :
Do you remember the last time we had a problem with this image: https://github.com/orgs/apache/packages?tab=packages&q=airflow-openldap ?
Me neither.
And we had plenty of similar problems with those images from time time to time when they were pulled from dockerhub. It's literaly 0 overhead - I pushed it once 11 months ago and did not touch it since. I think the one time effort of moving stuff to GitHub (either as separate repo or vendor-in) is certainly worth it over the long period of time.
The nice thing of keeping everything in GitHub is that when it fails, it generally fails in the way that many things do not work (including actions or CI in general). So when something breaks in GitHub people generally are aware of it (and more often than not they simply cannot even submit or run their CI jobs).
And if we see intermittent errors like that we have enterprise level support with them via Apache agreement - they either commented with workarounds or fixed many isues I raised to them already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #24037:
URL: https://github.com/apache/airflow/issues/24037#issuecomment-1144981070
It doesn't break our released versions thankfully (helm vendors it in the tarball already), but it does hose our CI and main for sure. +1 to vendoring it. Should we update to the newest version while we are at it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24037: Frequent failures of helm chart tests
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24037: Frequent failures of helm chart tests
URL: https://github.com/apache/airflow/issues/24037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org