You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2020/05/26 21:33:01 UTC

[spark] branch branch-2.4 updated: [SPARK-31819][K8S][DOCS][TESTS][2.4] Add a workaround for Java 8u251+/K8s 1.17 and update integration test cases

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new 48ba885  [SPARK-31819][K8S][DOCS][TESTS][2.4] Add a workaround for Java 8u251+/K8s 1.17 and update integration test cases
48ba885 is described below

commit 48ba885b0734165bff143c9d0a19e2b8ab63f42e
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue May 26 14:30:35 2020 -0700

    [SPARK-31819][K8S][DOCS][TESTS][2.4] Add a workaround for Java 8u251+/K8s 1.17 and update integration test cases
    
    ### What changes were proposed in this pull request?
    
    This PR aims to add a workaround `HTTP2_DISABLE=true` to the document and to update the K8s integration test.
    
    ### Why are the changes needed?
    
    SPARK-31786 reported fabric8 kubernetes-client library fails to talk K8s 1.17.x client on Java 8u251+ environment. It's fixed at Apache Spark 3.0.0 by upgrading the library, but it turns out that we can not use the same way in `branch-2.4` (https://github.com/apache/spark/pull/28625)
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. This will provide a workaround at the document and testing environment.
    
    ### How was this patch tested?
    
    This PR is irrelevant to Jenkins UT because it's only updating docs and integration tests. We need to the followings.
    - [x] Pass the Jenkins K8s IT with old JDK8 and K8s versions (https://github.com/apache/spark/pull/28638#issuecomment-633837179)
    - [x] Manually run K8s IT on K8s 1.17/Java 8u251+ with `export HTTP2_DISABLE=true`.
    
    **K8s v1.17.6 / JDK 1.8.0_252**
    ```
    KubernetesSuite:
    - Run SparkPi with no resources
    - Run SparkPi with a very long application name.
    - Use SparkLauncher.NO_RESOURCE
    - Run SparkPi with a master URL without a scheme.
    - Run SparkPi with an argument.
    - Run SparkPi with custom labels, annotations, and environment variables.
    - Run extraJVMOptions check on driver
    - Run SparkRemoteFileTest using a remote data file
    - Run SparkPi with env and mount secrets.
    - Run PySpark on simple pi.py example
    - Run PySpark with Python2 to test a pyfiles example
    - Run PySpark with Python3 to test a pyfiles example
    - Run PySpark with memory customization
    - Run in client mode.
    Run completed in 5 minutes, 7 seconds.
    Total number of tests run: 14
    Suites: completed 2, aborted 0
    Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0
    All tests passed.
    ```
    
    Closes #28638 from dongjoon-hyun/SPARK-31819.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 docs/index.md                                                        | 2 ++
 resource-managers/kubernetes/integration-tests/README.md             | 2 ++
 .../spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala      | 5 ++++-
 .../apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala    | 1 +
 4 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/docs/index.md b/docs/index.md
index 9a59097..52f1a5a 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -39,6 +39,8 @@ Note that support for Java 7, Python 2.6 and old Hadoop versions before 2.6.5 we
 Support for Scala 2.10 was removed as of 2.3.0. Support for Scala 2.11 is deprecated as of Spark 2.4.1
 and will be removed in Spark 3.0.
 
+For Java 8u251+, `HTTP2_DISABLE=true` and `spark.kubernetes.driverEnv.HTTP2_DISABLE=true` are required additionally for fabric8 `kubernetes-client` library to talk to Kubernetes clusters. This prevents `KubernetesClientException` when `kubernetes-client` library uses `okhttp` library internally.
+
 # Running the Examples and Shell
 
 Spark comes with several sample programs.  Scala, Java, Python and R examples are in the
diff --git a/resource-managers/kubernetes/integration-tests/README.md b/resource-managers/kubernetes/integration-tests/README.md
index b3863e6..20bd8b5 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -12,6 +12,8 @@ The simplest way to run the integration tests is to install and run Minikube, th
 
     dev/dev-run-integration-tests.sh
 
+For Java 8u251+, `HTTP2_DISABLE=true` and `spark.kubernetes.driverEnv.HTTP2_DISABLE=true` are required additionally for fabric8 `kubernetes-client` library to talk to Kubernetes clusters. This prevents `KubernetesClientException` when `kubernetes-client` library uses `okhttp` library internally.
+
 The minimum tested version of Minikube is 0.23.0. The kube-dns addon must be enabled. Minikube should
 run with a minimum of 3 CPUs and 4G of memory:
 
diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala
index c8bd584..d1508e9 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala
+++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala
@@ -16,9 +16,11 @@
  */
 package org.apache.spark.deploy.k8s.integrationtest
 
-import org.scalatest.concurrent.Eventually
 import scala.collection.JavaConverters._
 
+import io.fabric8.kubernetes.api.model.EnvVar
+import org.scalatest.concurrent.Eventually
+
 import org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.{k8sTestTag, INTERVAL, TIMEOUT}
 
 private[spark] trait ClientModeTestsSuite { k8sSuite: KubernetesSuite =>
@@ -66,6 +68,7 @@ private[spark] trait ClientModeTestsSuite { k8sSuite: KubernetesSuite =>
             .withName("spark-example")
             .withImage(image)
             .withImagePullPolicy("IfNotPresent")
+            .withEnv(new EnvVar("HTTP2_DISABLE", "true", null))
             .withCommand("/opt/spark/bin/run-example")
             .addToArgs("--master", s"k8s://https://kubernetes.default.svc")
             .addToArgs("--deploy-mode", "client")
diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index e6840ce..d893433 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -108,6 +108,7 @@ private[spark] class KubernetesSuite extends SparkFunSuite
       .set("spark.kubernetes.container.image", image)
       .set("spark.kubernetes.driver.pod.name", driverPodName)
       .set("spark.kubernetes.driver.label.spark-app-locator", appLocator)
+      .set("spark.kubernetes.driverEnv.HTTP2_DISABLE", "true")
       .set("spark.kubernetes.executor.label.spark-app-locator", appLocator)
     if (!kubernetesTestComponents.hasUserSpecifiedNamespace) {
       kubernetesTestComponents.createNamespace()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org