You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2022/07/09 13:04:51 UTC
[GitHub] [openwhisk] style95 opened a new issue, #5274: SocketTimeoutException in the Kubernetes environment.
style95 opened a new issue, #5274:
URL: https://github.com/apache/openwhisk/issues/5274
When we keep invoking an abnormal action, it could cause the `SocketTimeoutException` in the invoker and it makes the invoker no longer able to create pods.
## Environment details:
* Kubernetes
## Steps to reproduce the issue:
1. Create a black-box action with an abnormal image such as `centos:centos7` that does not have any kind of action proxy.
2. Keep invoking the action every 1s.
```bash
while :
do
wsk action invoke abnormal-image
sleep 1s
done
```
3. After around 1 hour, an invoker starts printing `SocketTimeoutException` even for normal actions as well.
```bash
[2021-04-05T10:38:20.573Z] [ERROR] [#tid_sid_invokerWarmup] [KubernetesClient] Failed create pod for 'wsk-blue-real-invoker-07-175029-prewarm-shellv2': class io.fabric8.kubernetes.client.KubernetesClientException (Caused by: java.net.SocketTimeoutException: timeout) - An error has occurred.; stacktrace: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:243)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:195)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:409)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:234)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:796)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:326)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:322)
at org.apache.openwhisk.core.containerpool.kubernetes.KubernetesClient.$anonfun$run$3(KubernetesClient.scala:167)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.openwhisk.core.containerpool.kubernetes.KubernetesClient.run(KubernetesClient.scala:166)
at org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainer$.create(KubernetesContainer.scala:76)
at org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainerFactory.createContainer(KubernetesContainerFactory.scala:82)
at org.apache.openwhisk.core.invoker.InvokerReactiveV2.$anonfun$childFactory$2(InvokerReactiveV2.scala:291)
at org.apache.openwhisk.core.invoker.InvokerReactiveV2.$anonfun$childFactory$2$adapted(InvokerReactiveV2.scala:291)
at org.apache.openwhisk.core.containerpool.v2.ContainerProxyV2$$anonfun$1.applyOrElse(ContainerProxyV2.scala:209)
at org.apache.openwhisk.core.containerpool.v2.ContainerProxyV2$$anonfun$1.applyOrElse(ContainerProxyV2.scala:197)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at akka.actor.FSM.processEvent(FSM.scala:707)
at akka.actor.FSM.processEvent$(FSM.scala:704)
at org.apache.openwhisk.core.containerpool.v2.ContainerProxyV2.processEvent(ContainerProxyV2.scala:147)
at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701)
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695)
at akka.actor.Actor.aroundReceive(Actor.scala:539)
at akka.actor.Actor.aroundReceive$(Actor.scala:537)
at org.apache.openwhisk.core.containerpool.v2.ContainerProxyV2.aroundReceive(ContainerProxyV2.scala:147)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
at akka.actor.ActorCell.invoke(ActorCell.scala:581)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
at akka.dispatch.Mailbox.run(Mailbox.scala:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: timeout
at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:656)
at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:664)
at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.java:398)
at okhttp3.internal.http2.Http2Codec$StreamFinishingSource.read(Http2Codec.java:205)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:439)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:233)
... 32 more
```
This means, if any malicious user keeps trying to invoke an abnormal action, it could break the system.
## Additional information you deem important:
I confirmed it does not happen anymore after upgrading the k8s client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] style95 closed issue #5274: SocketTimeoutException in the Kubernetes environment.
Posted by GitBox <gi...@apache.org>.
style95 closed issue #5274: SocketTimeoutException in the Kubernetes environment.
URL: https://github.com/apache/openwhisk/issues/5274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org