You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/10/21 08:45:13 UTC
[GitHub] [apisix] shareinto opened a new issue, #8144: bug: The data returned by k8s-apiserver watch api is truncated
shareinto opened a new issue, #8144:
URL: https://github.com/apache/apisix/issues/8144
### Current Behavior
I use apisix as the proxy of k8s-apiserver, when I use kubectl node-shell plugin（https://github.com/kvaps/kubectl-node-shell）, there will be an error of timed out wait for condition
```
k node-shell $NODE ──(Fri,Oct21)─┘
spawning "nsenter-uvxhk8" on "..."
error: timed out waiting for the condition
pod "nsenter-uvxhk8" deleted
```
It is normal if nginx ingress controller is used as a proxy
I tracked kubectl's requests and found that it uses k8s watch interface to discover pod status changes,like that
```
/api/v1/namespaces/default/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dtest&resourceVersion=0&timeout=9m8s&timeoutSeconds=548&watch=true
```
when proxy is apisix ,the response data was truncated ,like this:
```
{"type":"DELETED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/test","uid":"aa7b70dd-cf23-4f05-aff4-3563b65ee3e0","resourceVersion":"926306705","creationTimestamp":"2022-10-21T08:38:12Z","deletionTimestamp":"2022-10-21T08:38:22Z","deletionGracePeriodSeconds":0,"labels":{"run":"test"},"annotations":{"kubernetes.io/psp":"eks.privileged"},"managedFields":[{"manager":"kubectl-run","operation":"Update","apiVersion":"v1","time":"2022-10-21T08:38:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:run":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"test\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:stdin":{},"f:stdinOnce":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:tty":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSec
onds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2022-10-21T08:38:54Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.0.253.5\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-kgzrj","secret":{"secretName":"default-token-kgzrj","defaultMode":420}}],"containers":[{"name":"test","image":"alpine","args":["sh"],"resources":{},"volumeMounts":[{"name":"default-token-kgzrj","readOnly":true,"mountPath":"/var/run/secrets
/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","stdin":true,"stdinOnce":true,"tty":true}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"ip-10-0-253-23.us-east-2.compute.internal","securityContext":{},"imagePullSecrets":[{"name":"image-pull-secret"}],"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-10-21T08:38:12Z"},{"type":"Ready","status":"False","lastProbeTime":nu
ll,"lastTransitionTime":"2022-10-21T08:38:54Z","reason":"ContainersNotReady","message":"containers with unready status: [test]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2022-10-21T08:38:54Z","reason":"ContainersNotReady","message":"containers with unready status: [test]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-10-21T08:38:12Z"}],"hostIP":"10.0.253.23","podIP":"10.0.253.5","podIPs":[{"ip":"10.0.253.5"}],"startTime":"2022-10-21T08:38:12Z","containerStatuses":[{"name":"test","state":{"terminated":{"exitCode":137,"reason":"Error","startedAt":"2022-10-21T08:38:19Z","finishedAt":"2022-10-21T08:38:52Z","containerID":"docker://c9e1261fec4914138cd2aac0348c25f757b7468cab401db78075fcdac75f7dec"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2022-10-21T08:38:13Z","f
```
And this doesn't happen on nginx ingress controller
So is this a bug, or is there something wrong with the configuration, please help
### Expected Behavior
kubectl node-shell spawn normaly
### Error Logs
```
k node-shell $NODE ──(Fri,Oct21)─┘
spawning "nsenter-uvxhk8" on "..."
error: timed out waiting for the condition
pod "nsenter-uvxhk8" deleted
```
### Steps to Reproduce
1, use apisix as k8s-apiserver proxy
2. use kubectl node-shell plugin spawn a node
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shareinto commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286719520
> > when proxy is apisix ,the response data was truncated ,like this:
>
> So do you mean the output of `kubectl` is truncated after you use APISIX as the proxy?
not the output of kubectl , is the response of the k8s watch api, and i use nginx ingress controller as the proxy ,this situation does not happen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286705854
@shareinto Please provide environment information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286706947
> when proxy is apisix ,the response data was truncated ,like this:
So do you mean the output of `kubectl` is truncated after you use APISIX as the proxy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286667164
Hi:
Maybe this is just a log truncation?
I seem to remember that the default log length is 4k , and your output is pretty close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shareinto commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286741739
Transfer-Encoding: chunked - there is no content length specified, maybe this lead to truncated happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fzp11 commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
fzp11 commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1310272034
我再使用 https http1.1 Transfer-Encoding: chunked 的时候也有这个问题，响应提被截断，没有正常返回
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shareinto commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286734763
```
10.0.130.127 - - [21/Oct/2022:09:55:28 +0000] XXX "GET /k8s/clusters/local/api/v1/namespaces/default/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dnsenter-q59hy6&resourceVersion=926371716&timeout=7m44s&timeoutSeconds=464&watch=true HTTP/2.0" 200 3907 59.753 "-" "kubectl/v1.23.6 (linux/amd64) kubernetes/ad33385" 10.0.82.185:80 200 59.754 "http://xxxx"
```
this is the watch api log ,and it took about 60s, when i use nginx ingress controller as proxy ,this api took about 10s.
```
k run -h
...
--pod-running-timeout=1m0s: The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one
pod is running
...
```
and the 60s is the default value of pod-running-timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: The data returned by k8s-apiserver watch api is truncated [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1806771418
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shareinto commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1288392833
```
"plugins": {
"proxy-control": {
"disable": false,
"request_buffering": false
}
}
```
@tokers it doesn't seem to work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1288767209
@shareinto I made a mistake. The proxy-control plugin cannot modify the proxy_buffering. You may have to change it through the custom Nginx configuration: https://apisix.apache.org/docs/apisix/customize-nginx-configuration/.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1287990795
> Transfer-Encoding: chunked - there is no content length specified, maybe this lead to truncated happen?
I don't think so. Watch mechanism relies on the chunk feature in http/1.1.
I guess it's related to the buffering of apisix. Try to diable the proxy_buffering feature. You can refer to the proxy_control plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shareinto commented on issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1286723776
kubectl uses the watch api to monitor the change of the pod status, and the watch api is truncated when it returns, so kubectl cannot perceive the change of the pod status, so the timed out for condition error occurs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: The data returned by k8s-apiserver watch api is truncated [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8144: bug: The data returned by k8s-apiserver watch api is truncated
URL: https://github.com/apache/apisix/issues/8144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: The data returned by k8s-apiserver watch api is truncated [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8144:
URL: https://github.com/apache/apisix/issues/8144#issuecomment-1782646376
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org