[jira] [Created] (SPARK-40954) Kubernetes integration tests stuck forever on Mac M1 with Minikube + Docker

["Anton Ippolitov (Jira)" <ji...@apache.org>]

Anton Ippolitov created SPARK-40954:
---------------------------------------

             Summary: Kubernetes integration tests stuck forever on Mac M1 with Minikube + Docker
                 Key: SPARK-40954
                 URL: https://issues.apache.org/jira/browse/SPARK-40954
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes, Tests
    Affects Versions: 3.3.1
         Environment: MacOS 12.6 (Mac M1)

Minikube 1.27.1

Docker 20.10.17
            Reporter: Anton Ippolitov


h2. Description

I tried running Kubernetes integration tests with the Minikube backend (+ Docker driver) from commit c26d99e3f104f6603e0849d82eca03e28f196551 on Spark's master branch. I ran them with the following command:

 
{code:java}
mvn integration-test -am -pl :spark-kubernetes-integration-tests_2.12 \
                        -Pkubernetes -Pkubernetes-integration-tests \
                        -Phadoop-3 \
                        -Dspark.kubernetes.test.imageTag=MY_IMAGE_TAG_HERE \
                        -Dspark.kubernetes.test.imageRepo=docker.io/kubespark \
                        -Dspark.kubernetes.test.namespace=spark \
                        -Dspark.kubernetes.test.serviceAccountName=spark \
                        -Dspark.kubernetes.test.deployMode=minikube  {code}
However the test suite got stuck literally for hours on my machine. 

 
h2. Investigation

I ran {{jstack}} on the process that was running the tests and saw that it was stuck here:

 
{noformat}
"ScalaTest-main-running-KubernetesSuite" #1 prio=5 os_prio=31 tid=0x00007f78d580b800 nid=0x2503 runnable [0x0000000304749000]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:255)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    - locked <0x000000076c0b6f40> (a java.lang.UNIXProcess$ProcessPipeInputStream)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    - locked <0x000000076c0bb410> (a java.io.InputStreamReader)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    - locked <0x000000076c0bb410> (a java.io.InputStreamReader)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at org.apache.spark.deploy.k8s.integrationtest.ProcessUtils$.$anonfun$executeProcess$2(ProcessUtils.scala:45)
    at org.apache.spark.deploy.k8s.integrationtest.ProcessUtils$.$anonfun$executeProcess$2$adapted(ProcessUtils.scala:45)
    at org.apache.spark.deploy.k8s.integrationtest.ProcessUtils$$$Lambda$322/20156341.apply(Unknown Source)
    at org.apache.spark.deploy.k8s.integrationtest.Utils$.tryWithResource(Utils.scala:49)
    at org.apache.spark.deploy.k8s.integrationtest.ProcessUtils$.executeProcess(ProcessUtils.scala:45)
    at org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.executeMinikube(Minikube.scala:103)
    at org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.minikubeServiceAction(Minikube.scala:112)
    at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.$anonfun$getServiceUrl$1(DepsTestsSuite.scala:281)
    at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite$$Lambda$611/1461360262.apply(Unknown Source)
    at org.scalatest.enablers.Retrying$$anon$4.makeAValiantAttempt$1(Retrying.scala:184)
    at org.scalatest.enablers.Retrying$$anon$4.tryTryAgain$2(Retrying.scala:196)
    at org.scalatest.enablers.Retrying$$anon$4.retry(Retrying.scala:226)
    at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:313)
    at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:312)
    at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:457)
    at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.getServiceUrl(DepsTestsSuite.scala:278)
    at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.tryDepsTest(DepsTestsSuite.scala:325)
    at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.$anonfun$$init$$1(DepsTestsSuite.scala:160)
    at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite$$Lambda$178/1750286943.apply$mcV$sp(Unknown Source)
[...]{noformat}
 So the issue is coming from {{DepsTestsSuite}} when it is setting up {{{}minio{}}}. After [creating the minio StatefulSet and Service|https://github.com/apache/spark/blob/5ea2b386eb866e20540660cdb6ed43792cb29969/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala#L85], it [executes|https://github.com/apache/spark/blob/5ea2b386eb866e20540660cdb6ed43792cb29969/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala#L280-L281] the '{{{}minikube service -n spark minio-s3 --url'{}}} command. It then gets stuck in {{ProcessUtils}} while reading {{{}minikube{}}}'s stdout [here.|https://github.com/apache/spark/blob/c8b7a09d39bdbda1502a7580fe2b54b7cb0ac4e3/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ProcessUtils.scala#L44-L50]

I then ran the same command from my shell and confirmed that it never returns until a CTRL+C:
{noformat}
$ minikube service -n spark minio-s3 --url
http://127.0.0.1:63114
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

<COMMAND IS STILL RUNNING HERE>{noformat}
So it looks like it's the normal behaviour for the 'minikube service' command on Mac with the Docker driver: it needs to keep an open tunnel. I had a quick look at Minikube's source code and it seems to be happening here: [https://github.com/kubernetes/minikube/blob/abed8b7d347ae15fe9c0acd91b5b49b3b6494a53/cmd/minikube/cmd/service.go#L154]

It also seems to be confirmed by the docs: [https://minikube.sigs.k8s.io/docs/handbook/accessing/] 

So the code which reads from stdout hangs indefinitely because of that.

 

I am not sure what would be the best solution here. I think ideally, we should run the  'minikube service' command, then retrieve the URL without blocking but at the same time we should make sure to leave the command running. When the {{DepsTestsSuite}} terminates, we shouldn't forget to terminate the minikube too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org