You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shane Knapp (Jira)" <ji...@apache.org> on 2021/05/17 23:13:00 UTC
[jira] [Resolved] (SPARK-34738) Upgrade Minikube and kubernetes and
move to docker 'virtualization' layer
[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shane Knapp resolved SPARK-34738.
---------------------------------
Resolution: Fixed
k8s tests are passing! marking this as 'resolved'.
> Upgrade Minikube and kubernetes and move to docker 'virtualization' layer
> -------------------------------------------------------------------------
>
> Key: SPARK-34738
> URL: https://issues.apache.org/jira/browse/SPARK-34738
> Project: Spark
> Issue Type: Task
> Components: jenkins, Kubernetes
> Affects Versions: 3.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Shane Knapp
> Priority: Major
> Attachments: integration-tests.log
>
>
> [~shaneknapp] as we discussed [on the mailing list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html] Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
> [Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new method to configure the kubernetes client. Thanks in advance to use it for testing on the Jenkins after the Minikube version is updated.
>
> Added by Shane:
> we also need to move from the kvm2 virtualization layer to docker. docker is a recommended driver w/the latest versions of minikube, and this will allow devs to more easily recreate the minikube/k8s env on their local workstations and run the integration tests in an identical environment as jenkins.
> the TL;DR is that upgrading to docker works, except that the PV integration tests are failing due to a couple of possible reasons:
> 1) the 'spark-kubernetes-driver' isn't properly being loaded (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
> 2) during the PV test run, the error message 'Given path (/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in the logs. however, the mk cluster *does* mount successfully to the local bare-metal filesystem *and* if i 'minikube ssh' in to it, i can see the mount and read/write successfully to it (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312548&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312548)
> i could really use some help, and if it's useful, i can create some local accounts manually and allow ssh access for a couple of people to assist me.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org