You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lk_spark <lk...@163.com> on 2018/04/08 03:15:51 UTC

spark2.3 on kubernets

hi,all:
   I am trying spark on k8s with Pi sample.
   I got error with driver :

   2018-04-08 03:08:40 INFO  SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated
 at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2747)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
 at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
 at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
 at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
 at scala.Option.getOrElse(Option.scala:121)
 at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
 at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
 at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [spark-pi-d439505012c5362ea6d26ea19df58bc1-driver]  in namespace: [default]  failed.
 at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
 at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:228)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:184)
 at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.<init>(KubernetesClusterSchedulerBackend.scala:70)
 at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:120)
 at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2741)
 ... 8 more
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname kubernetes.default.svc not verified:
    certificate: sha256/Buyjy7p4sLKA7rLJqT0N/Cr3sM82fyi3pmx+eSht0eI=
    DN: CN=kube-apiserver, OU=Kubernetes-manual, O=Kubernetes, L=Taipei, ST=Taipei, C=TW
    subjectAltNames: [10.96.0.1, 192.168.0.140, 127.0.0.1, kubernetes.default]
 at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:290)
 at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251)
 at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151)
 at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195)
 at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
 at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
 at okhttp3.RealCall.execute(RealCall.java:69)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:312)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:295)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:783)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:217)
 ... 12 more

It seams like can't connect with k8s's api server ? I don't know what to do . my commond for test is :

./bin/spark-submit \
    --master k8s://https://192.168.0.140:6443 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.node.selector.spark=true \
    --conf spark.kubernetes.authenticate.driver.clientKeyFile=/etc/kubernetes/pki/apiserver.pem \
    --conf spark.kubernetes.container.image=dpaas/apache/spark:2.3.0-v1 \
    local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar

2018-04-08


lk_spark

Re: spark2.3 on kubernets

Posted by lk_spark <lk...@163.com>.
resolved. need to add "kubernetes.default.svc" to k8s api server TLS config.

2018-04-08 



lk_spark 



发件人："lk_spark"<lk...@163.com>
发送时间：2018-04-08 11:15
主题：spark2.3 on kubernets
收件人："user"<us...@spark.apache.org>
抄送：

hi,all:
   I am trying spark on k8s with Pi sample.
   I got error with driver :

   2018-04-08 03:08:40 INFO  SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated
 at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2747)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
 at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
 at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
 at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
 at scala.Option.getOrElse(Option.scala:121)
 at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
 at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
 at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [spark-pi-d439505012c5362ea6d26ea19df58bc1-driver]  in namespace: [default]  failed.
 at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
 at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:228)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:184)
 at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.<init>(KubernetesClusterSchedulerBackend.scala:70)
 at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:120)
 at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2741)
 ... 8 more
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname kubernetes.default.svc not verified:
    certificate: sha256/Buyjy7p4sLKA7rLJqT0N/Cr3sM82fyi3pmx+eSht0eI=
    DN: CN=kube-apiserver, OU=Kubernetes-manual, O=Kubernetes, L=Taipei, ST=Taipei, C=TW
    subjectAltNames: [10.96.0.1, 192.168.0.140, 127.0.0.1, kubernetes.default]
 at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:290)
 at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251)
 at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151)
 at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195)
 at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
 at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
 at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
 at okhttp3.RealCall.execute(RealCall.java:69)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:312)
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:295)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:783)
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:217)
 ... 12 more

It seams like can't connect with k8s's api server ? I don't know what to do . my commond for test is :

./bin/spark-submit \
    --master k8s://https://192.168.0.140:6443 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.node.selector.spark=true \
    --conf spark.kubernetes.authenticate.driver.clientKeyFile=/etc/kubernetes/pki/apiserver.pem \
    --conf spark.kubernetes.container.image=dpaas/apache/spark:2.3.0-v1 \
    local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar

2018-04-08


lk_spark