You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/10 21:13:39 UTC
[GitHub] [airflow] dimberman opened a new pull request #9220: Get all logs on failure
dimberman opened a new pull request #9220:
URL: https://github.com/apache/airflow/pull/9220
uses `--all-containers=true` flag to get all k8s logs
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Target Github ISSUE in description if exists
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642504818
Hey @dimberman @ashb -> I see there were some problems with Kubernetes on CI . I tried to reproduce it locally https://github.com/apache/airflow/blob/master/TESTING.rst#typical-testing-pattern-for-kubernetes-tests but of course app can be deployed there without any problems.
I have started long weekend and have little access to computer now - so I hope you will solve it eventually but for what it can be worth - we already had a function that was sending all the logs from kind (kind has the useful "kind export logs" feature for that but it was not wired to the scripts after the recent refactor. I added it to send logs always for now in #9228 .
I planned to use Github Action artifacts instead of file.io - so maybe you can do just that as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642515141
Can I help somehow ? I have a little time before I get to my car in 400 km trip :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#discussion_r438468236
##########
File path: scripts/ci/libraries/_kind.sh
##########
@@ -365,19 +365,23 @@ function apply_kubernetes_resources() {
function dump_kubernetes_logs() {
POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
--cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+ kubectl get all
+ kubectl get all --namespace kube-system
+ echo "------coredns logs-------"
+ DNS_POD=$(kubectl get pods -o go-template -n kube-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+ --cluster "${KUBECTL_CLUSTER_NAME}" | grep dns | head -1)
Review comment:
You forgot the logs command :dagger:
Also an easier way for this
```
```suggestion
kubectl -n kube-system logs -l k8s-app=kube-dns --all-containers --cluster "${KUBECTL_CLUSTER_NAME}"
```
(Even with coredns, the k8s-app label is still kube-dns)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642695089
Giving up for now - not getting anywhere with why the git-sync tests are failing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#discussion_r438468236
##########
File path: scripts/ci/libraries/_kind.sh
##########
@@ -365,19 +365,23 @@ function apply_kubernetes_resources() {
function dump_kubernetes_logs() {
POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
--cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+ kubectl get all
+ kubectl get all --namespace kube-system
+ echo "------coredns logs-------"
+ DNS_POD=$(kubectl get pods -o go-template -n kube-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+ --cluster "${KUBECTL_CLUSTER_NAME}" | grep dns | head -1)
Review comment:
You forgot the logs command :dagger:
Also an easier way for this
```suggestion
kubectl -n kube-system logs -l k8s-app=kube-dns --all-containers --cluster "${KUBECTL_CLUSTER_NAME}"
```
(Even with coredns, the k8s-app label is still kube-dns)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#discussion_r438465611
##########
File path: scripts/ci/libraries/_kind.sh
##########
@@ -365,19 +365,23 @@ function apply_kubernetes_resources() {
function dump_kubernetes_logs() {
POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
--cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+ kubectl get all
+ kubectl get all --namespace kube-system
+ echo "------coredns logs-------"
+ DNS_POD=$(kubectl get pods -o go-template -n kube-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+ --cluster "${KUBECTL_CLUSTER_NAME}" | grep dns | head -1)
echo "------- pod description -------"
kubectl describe pod "${POD}" --cluster "${KUBECTL_CLUSTER_NAME}"
- echo "------- webserver init container logs - init -------"
- kubectl logs "${POD}" -c init --cluster "${KUBECTL_CLUSTER_NAME}" || true
- if [[ "${KUBERNETES_MODE}" == "git" ]]; then
- echo "------- webserver init container logs - git-sync-clone -------"
- kubectl logs "${POD}" -c git-sync-clone --cluster "${KUBECTL_CLUSTER_NAME}" || true
- fi
- echo "------- webserver logs -------"
- kubectl logs "${POD}" -c webserver --cluster "${KUBECTL_CLUSTER_NAME}" || true
- echo "------- scheduler logs -------"
- kubectl logs "${POD}" -c scheduler --cluster "${KUBECTL_CLUSTER_NAME}" || true
- echo "--------------"
+ echo "------- airflow pod logs -------"
+ kubectl logs "${POD}" --all-containers=true || true
+ PG_POD=POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+ --cluster "${KUBECTL_CLUSTER_NAME}" | grep postgres | head -1)
+ echo "------- postgres pod description -------"
+ kubectl describe pod "${PG_POD}" --cluster "${KUBECTL_CLUSTER_NAME}"
+ echo "------- postgres pod logs -------"
+ kubectl logs "${PG_POD}" --all-containers=true || true
Review comment:
Something not quite right here:
```
------- postgres pod description -------
Error from server (NotFound): pods "POD=postgres-airflow-685c8f6684-6nxnl" not found
###########################################################################################
EXITING WITH STATUS CODE 1
###########################################################################################
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#discussion_r438465926
##########
File path: scripts/ci/libraries/_kind.sh
##########
@@ -365,19 +365,23 @@ function apply_kubernetes_resources() {
function dump_kubernetes_logs() {
POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
--cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+ kubectl get all
+ kubectl get all --namespace kube-system
+ echo "------coredns logs-------"
+ DNS_POD=$(kubectl get pods -o go-template -n kube-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+ --cluster "${KUBECTL_CLUSTER_NAME}" | grep dns | head -1)
echo "------- pod description -------"
kubectl describe pod "${POD}" --cluster "${KUBECTL_CLUSTER_NAME}"
- echo "------- webserver init container logs - init -------"
- kubectl logs "${POD}" -c init --cluster "${KUBECTL_CLUSTER_NAME}" || true
- if [[ "${KUBERNETES_MODE}" == "git" ]]; then
- echo "------- webserver init container logs - git-sync-clone -------"
- kubectl logs "${POD}" -c git-sync-clone --cluster "${KUBECTL_CLUSTER_NAME}" || true
- fi
- echo "------- webserver logs -------"
- kubectl logs "${POD}" -c webserver --cluster "${KUBECTL_CLUSTER_NAME}" || true
- echo "------- scheduler logs -------"
- kubectl logs "${POD}" -c scheduler --cluster "${KUBECTL_CLUSTER_NAME}" || true
- echo "--------------"
+ echo "------- airflow pod logs -------"
+ kubectl logs "${POD}" --all-containers=true || true
+ PG_POD=POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
Review comment:
```suggestion
PG_POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on pull request #9220: Get all logs on failure
Posted by GitBox <gi...@apache.org>.
dimberman commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642270936
@potiuk @ashb this should help us figure out why the init-db container is failing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on pull request #9220: Get all logs on failure
Posted by GitBox <gi...@apache.org>.
dimberman commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642311700
@ashb yes was just thinking that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb closed pull request #9220:
URL: https://github.com/apache/airflow/pull/9220
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642605888
Not needed now for #9230, but using this to investigate the git init problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #9220: Get all logs on failure
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642311498
@dimberman Can you also make it dump the logs/status of postgres, and I guess kubedns/coredns/whatever DNS is actually in use?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9220: Get all pod logs on Kube CI failure
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9220:
URL: https://github.com/apache/airflow/pull/9220#issuecomment-642514472
BTW. Did you figure out why it suddenly stopped working - I see that you modified coredns configuration again - something that we've done long time ago when kind was starting from the container
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org