You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Pralabh Kumar <pr...@gmail.com> on 2022/03/16 13:10:50 UTC

Skip single integration test case in Spark on K8s

Hi Spark team

I am running Spark kubernetes integration test suite on cloud.

build/mvn install \

-f  pom.xml \

-pl resource-managers/kubernetes/integration-tests -am -Pscala-2.12
-Phadoop-3.1.1 -Phive -Phive-thriftserver -Pyarn -Pkubernetes
-Pkubernetes-integration-tests \

-Djava.version=8 \

-Dspark.kubernetes.test.sparkTgz=<file> \

-Dspark.kubernetes.test.imageTag=<> \

-Dspark.kubernetes.test.imageRepo=< <http://reg.visa.com/>repo> \

-Dspark.kubernetes.test.deployMode=cloud \

-Dtest.include.tags=k8s \

-Dspark.kubernetes.test.javaImageTag=<tag> \

-Dspark.kubernetes.test.namespace=<name> \

-Dspark.kubernetes.test.serviceAccountName=spark \

-Dspark.kubernetes.test.kubeConfigContext=<> \

-Dspark.kubernetes.test.master=<> \

-Dspark.kubernetes.test.jvmImage=<> \

-Dspark.kubernetes.test.pythonImage=<> \

-Dlog4j.logger.org.apache.spark=DEBUG



I am successfully able to run some test cases and some are failing . For
e.g "Run SparkRemoteFileTest using a Remote data file" in KuberneterSuite
is failing.


Is there a way to skip running some of the test cases ?.



Please help me on the same.


Regards

Pralabh Kumar

Re: Skip single integration test case in Spark on K8s

Posted by Dongjoon Hyun <do...@gmail.com>.

-user@spark

For cloud backend, you need to exclude minikube specific tests and
local-only test (SparkRemoteFileTest).

    -Dtest.exclude.tags=minikube,local

You can find more options including SBT commands here.


https://github.com/apache/spark/tree/master/resource-managers/kubernetes/integration-tests

Dongjoon.


On Wed, Mar 16, 2022 at 6:11 AM Pralabh Kumar <pr...@gmail.com>
wrote:

> Hi Spark team
>
> I am running Spark kubernetes integration test suite on cloud.
>
> build/mvn install \
>
> -f  pom.xml \
>
> -pl resource-managers/kubernetes/integration-tests -am -Pscala-2.12
> -Phadoop-3.1.1 -Phive -Phive-thriftserver -Pyarn -Pkubernetes
> -Pkubernetes-integration-tests \
>
> -Djava.version=8 \
>
> -Dspark.kubernetes.test.sparkTgz=<file> \
>
> -Dspark.kubernetes.test.imageTag=<> \
>
> -Dspark.kubernetes.test.imageRepo=< <http://reg.visa.com/>repo> \
>
> -Dspark.kubernetes.test.deployMode=cloud \
>
> -Dtest.include.tags=k8s \
>
> -Dspark.kubernetes.test.javaImageTag=<tag> \
>
> -Dspark.kubernetes.test.namespace=<name> \
>
> -Dspark.kubernetes.test.serviceAccountName=spark \
>
> -Dspark.kubernetes.test.kubeConfigContext=<> \
>
> -Dspark.kubernetes.test.master=<> \
>
> -Dspark.kubernetes.test.jvmImage=<> \
>
> -Dspark.kubernetes.test.pythonImage=<> \
>
> -Dlog4j.logger.org.apache.spark=DEBUG
>
>
>
> I am successfully able to run some test cases and some are failing . For
> e.g "Run SparkRemoteFileTest using a Remote data file" in KuberneterSuite
> is failing.
>
>
> Is there a way to skip running some of the test cases ?.
>
>
>
> Please help me on the same.
>
>
> Regards
>
> Pralabh Kumar
>