You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Gnana Kumar <gn...@gmail.com> on 2022/12/08 16:30:50 UTC
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Hi There,
I have been trying to follow the steps as per above url but below command
does not successfully create the binary distribution of spark with volcano.
./dev/make-distribution.sh --tgz --name with-volcano
-Pkubernetes,volcano,hadoop-3
Has anyone tried it and it works ? please confirm.
Thanks
Gnana
On Tue, Nov 29, 2022 at 10:16 PM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> MartinG have wrote a good howto on this
>
>
> https://martin-grigorov.medium.com/apache-spark-snapshot-on-kubernetes-178592613408
>
>
> https://martin-grigorov.medium.com/native-integration-between-apache-spark-and-volcano-kubernetes-scheduler-488f54dbbab3
>
>
> tir. 29. nov. 2022 kl. 16:25 skrev Gnana Kumar <gn...@gmail.com>:
>
>> Any updates on this please ?
>>
>> On Mon, Nov 28, 2022 at 11:58 AM Gnana Kumar <gn...@gmail.com>
>> wrote:
>>
>>> Yes, while building the binary dist, I have used the -Pvolcano option
>>> but for the docker build , do we have any options to use ?
>>>
>>> >docker build -t gnana_volcano_scheduler_snapshot -f
>>> kubernetes/dockerfiles/spark/Dockerfile .
>>>
>>> Because my understanding is that the binary dist will have all jars
>>> which should be packed within the container.
>>>
>>> Please let us know how to build docker image with volcano options.
>>>
>>> Thanks
>>> Gnana
>>>
>>> On Sun, Nov 27, 2022 at 3:41 PM Denis Bolshakov <
>>> bolshakov.denis@gmail.com> wrote:
>>>
>>>> I am talking about docker image, not about your local build. Could you
>>>> please make sure that docker image has a spark distro with volcano support?
>>>>
>>>> вс, 27 нояб. 2022 г., 13:09 Gnana Kumar <gn...@gmail.com>:
>>>>
>>>>> Yes. I have cloned the spark source 3.3.2.SNAPSHOT and then used below
>>>>> command to build the binary distribution with support of volcano.
>>>>>
>>>>> >dev/./make-distribution.sh -Denforcer.skip=true -DrecompileMode=all
>>>>> -Pkubernetes -Pvolcano -Pscala-2.12 -DskipTests
>>>>>
>>>>> Also please find the attached screenshot of spark-kubernetes JAR built
>>>>> from above command and if we unzip,it has the scala class -
>>>>> org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Thanks
>>>>> Gnana
>>>>>
>>>>> On Sun, Nov 27, 2022 at 3:23 PM Denis Bolshakov <
>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>
>>>>>> Does your driver docker image has a spark distro built with volcano
>>>>>> support?
>>>>>>
>>>>>> вс, 27 нояб. 2022 г., 11:55 Gnana Kumar <gn...@gmail.com>:
>>>>>>
>>>>>>> Actual error content is pasted below.
>>>>>>>
>>>>>>> Exception in thread "main"* java.lang.ClassNotFoundException:
>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep*
>>>>>>> at
>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>> at
>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>> at
>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>>>> at org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>> at
>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>> at
>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>> at
>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike.map(TraversableLike.scala:286)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike.map$(TraversableLike.scala:279)
>>>>>>> at
>>>>>>> scala.collection.AbstractTraversable.map(Traversable.scala:108)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:41)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:106)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:220)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:214)
>>>>>>> at
>>>>>>> org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2713)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:214)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:186)
>>>>>>> at org.apache.spark.deploy.SparkSubmit.org
>>>>>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>>> 22/11/27 08:35:12 INFO ShutdownHookManager: Shutdown hook called
>>>>>>>
>>>>>>> On Sun, Nov 27, 2022 at 2:11 PM Gnana Kumar <
>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>
>>>>>>>> Though my /opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar
>>>>>>>> has VolcanoFeatureStep.class sparksubmit says class is not found.
>>>>>>>>
>>>>>>>> Basically I'm looking for a working example of spark submit with
>>>>>>>> Volcano scheduler in Kubernetes.
>>>>>>>>
>>>>>>>> Please help me resolve this issue.
>>>>>>>>
>>>>>>>> ~/spark_3.3.2/bin/spark-submit \
>>>>>>>> --verbose \
>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>> --deploy-mode cluster \
>>>>>>>> --name sparkSampleApp2 \
>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>> --conf
>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>> \
>>>>>>>> --conf
>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>> \
>>>>>>>> --conf
>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>> \
>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>> --conf
>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>> \
>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>> --conf
>>>>>>>> spark.kubernetes.driver.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>> \
>>>>>>>> --conf
>>>>>>>> spark.kubernetes.executor.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>> \
>>>>>>>> --conf
>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>> --conf
>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>> --conf
>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>> --conf spark.kubernetes.file.upload.path=file:///tmp \
>>>>>>>> --jars
>>>>>>>> local:///opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar \
>>>>>>>>
>>>>>>>> local:///opt/spark/examples/jars/spark-analytics-application-1.0.jar
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> GK
>>>>>>>>
>>>>>>>> On Sun, Nov 27, 2022 at 2:03 PM Gnana Kumar <
>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Yes, I have installed it using the command below and volcano
>>>>>>>>> containers are coming up successfully in my GKE cluster.
>>>>>>>>>
>>>>>>>>> >kubectl apply -f
>>>>>>>>> https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/installer/volcano-development.yaml
>>>>>>>>>
>>>>>>>>> [image: image.png]
>>>>>>>>>
>>>>>>>>> However Spark submit cannot refer to the *VolcanoFeatureSetp
>>>>>>>>> class *and it says class has not found exception.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>> On Sun, Nov 27, 2022 at 2:53 AM Bjørn Jørgensen <
>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Have you installed volcano to the k8s cluster?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes
>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>> https://volcano.sh/en/docs/spark_on_volcano/
>>>>>>>>>>
>>>>>>>>>> lør. 26. nov. 2022 kl. 09:06 skrev Gnana Kumar <
>>>>>>>>>> gnana.kumar123@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> I have also attached jar which has VolcanoFeature.class.
>>>>>>>>>>>
>>>>>>>>>>> But when I run from SparkSubmit, it says class not found.
>>>>>>>>>>>
>>>>>>>>>>> # This is required for volcano
>>>>>>>>>>> ~/temp/spark_3.3.2/bin/spark-submit \
>>>>>>>>>>> --verbose \
>>>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>>>>> --deploy-mode cluster \
>>>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>>>> \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>> \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>> \
>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>>>> \
>>>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>>>> --conf
>>>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>>>>
>>>>>>>>>>> Please help me on how to run volcano using Spark Submit.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Nov 26, 2022 at 11:43 AM Gnana Kumar <
>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> HI Denis,
>>>>>>>>>>>>
>>>>>>>>>>>> This is how I have built it from the parent 'spark' dir.
>>>>>>>>>>>>
>>>>>>>>>>>> >mvn -Denforcer.skip=true -DrecompileMode=all -Pkubernetes
>>>>>>>>>>>> -Pvolcano -Pscala-2.12 -DskipTests clean package
>>>>>>>>>>>>
>>>>>>>>>>>> Also once the build is completed, the generated
>>>>>>>>>>>> spark-kubernetes...jar has VolcanoFeature.class but when this class name is
>>>>>>>>>>>> referred in Spark Submit ,it gives the class not found exception.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gnana
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Nov 25, 2022 at 1:43 PM Denis Bolshakov <
>>>>>>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think volcano is not enabled in default build, you have to
>>>>>>>>>>>>> provide a maven profile by adding -Pvolcano
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, 24 Nov 2022 at 19:09, Gnana Kumar <
>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any issue SNAPSHOT jars ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 24, 2022 at 9:33 PM Gnana Kumar <
>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi There,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have cloned spark 3.3.2 snapshot source code where Volcano
>>>>>>>>>>>>>>> feature is present and then I built the binary distribution for the same.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However when I tried to use this spark 3.3.2 SNAPSHOT dist
>>>>>>>>>>>>>>> with Volcano but I have got class not found exception.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Exception in thread "main" java.lang.ClassNotFoundException:
>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>>>>>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any help to resolve this issue please ? Basically I would
>>>>>>>>>>>>>>> like to use Volcano from Spark 3.3.2 in Kubernetes cluster.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> //with Best Regards
>>>>>>>>>>>>> --Denis Bolshakov
>>>>>>>>>>>>> e-mail: bolshakov.denis@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gnana
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>> Norge
>>>>>>>>>>
>>>>>>>>>> +47 480 94 297
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks
>>>>>>>> Gnana
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Gnana
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Gnana
>>>>>
>>>>
>>>
>>> --
>>> Thanks
>>> Gnana
>>>
>>
>>
>> --
>> Thanks
>> Gnana
>>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
--
Thanks
Gnana
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Bjørn Jørgensen <bj...@gmail.com>.
You have not build docker images to deploy on K8S
Running Spark on Kubernetes
<https://spark.apache.org/docs/latest/running-on-kubernetes.html>
--conf spark.kubernetes.container.image=<spark-image> \
søn. 18. des. 2022 kl. 17:45 skrev Gnana Kumar <gn...@gmail.com>:
> Also I have added the jar path in Spark Submit
>
> --jars local:///opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar
>
> On Sun, Dec 18, 2022 at 10:11 PM Gnana Kumar <gn...@gmail.com>
> wrote:
>
>> Here is the screenshot of Spark-Kubernetes jar containing
>> VolcanoFeatureStep.
>>
>> Not sure why is the spark-submit not able to load the class ?
>>
>> [image: image.png]
>>
>> On Sun, Dec 18, 2022 at 9:51 PM Gnana Kumar <gn...@gmail.com>
>> wrote:
>>
>>> I was able to build the docker image from spark v3.3.2 but when I run
>>> Spark-Submit , it gives me an exception which I have pasted down below.
>>>
>>> I'm not sure where docker build really includes the
>>> spark-kubernetes-3.3.2-SNAPSHOT.jar in docker image.
>>>
>>> ~/spark/bin/spark-submit \
>>> --verbose \
>>> --class com.demo.spark.SpringBootStarter \
>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>> --deploy-mode cluster \
>>> --name sparkSampleApp2 \
>>> --conf spark.kubernetes.namespace=default \
>>> --conf spark.network.timeout=300 \
>>> --conf spark.executor.instances=2 \
>>> --conf spark.kubernetes.scheduler.name=volcano \
>>> --conf
>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=$HOME/spark/podgroup-template.yaml
>>> \
>>> --conf
>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>> \
>>> --conf
>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>> \
>>>
>>>
>>> Exception in thread "main" java.lang.ClassNotFoundException:
>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>> at
>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>
>>> Thanks
>>> Gnana
>>>
>>> On Sun, Dec 18, 2022 at 7:43 PM Gnana Kumar <gn...@gmail.com>
>>> wrote:
>>>
>>>> Any help on Spark Docker build please ?
>>>>
>>>>
>>>> On Sat, Dec 17, 2022 at 8:34 PM Gnana Kumar <gn...@gmail.com>
>>>> wrote:
>>>>
>>>>> Now when I have tried to build the docker image using below commands
>>>>> ,I got the below error.
>>>>>
>>>>> BASE_OS="openeuler/openeuler:20.03-lts-sp3"
>>>>> SPARK_VERSION="3.3.2-SNAPSHOT"
>>>>> SCALA_VERSION="scala_2.12"
>>>>> DOCKERIMAGETAG="11-jre-slim"
>>>>>
>>>>> cd $SPARK_HOME
>>>>>
>>>>> ./bin/docker-image-tool.sh \
>>>>> -t ${SPARK_VERSION}-${SCALA_VERSION}-${DOCKERIMAGETAG} \
>>>>> -b java_image_tag=$DOCKERIMAGETAG \
>>>>> -p ./dist/kubernetes/dockerfiles/spark/Dockerfile \
>>>>> build
>>>>>
>>>>> *Error:*
>>>>> Step 5/18 : COPY jars /opt/spark/jars
>>>>> COPY failed: file not found in build context or excluded by
>>>>> .dockerignore: stat jars: file does not exist
>>>>> Failed to build PySpark Docker image, please refer to Docker build
>>>>> output for details.
>>>>>
>>>>> Thanks
>>>>> Gnana
>>>>>
>>>>> On Sat, Dec 17, 2022 at 9:37 AM Gnana Kumar <gn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Atlast it worked when I have tried with below config.
>>>>>>
>>>>>> -Xmx10g -Xss8g
>>>>>>
>>>>>> On Sat, Dec 17, 2022 at 8:59 AM Gnana Kumar <gn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have tried with several configurations like
>>>>>>>
>>>>>>> Xss 8m,16m,32m,64m,4gb etc
>>>>>>>
>>>>>>> But it is the same stackoverflow error.
>>>>>>>
>>>>>>> On Fri, Dec 16, 2022 at 11:08 PM Bjørn Jørgensen <
>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>
>>>>>>>> I use java 17 to build this.
>>>>>>>>
>>>>>>>> Are there any reasons why you have to build spark yourself? Can't
>>>>>>>> you start from spark 3.3.1 tar file and build a docker image from there?
>>>>>>>>
>>>>>>>> fre. 16. des. 2022 kl. 18:26 skrev Gnana Kumar <
>>>>>>>> gnana.kumar123@gmail.com>:
>>>>>>>>
>>>>>>>>> I have opened the pom file under cd ~/spark/sql/catalyst$ vi
>>>>>>>>> pom.xml and increases from "-Xss4m" to "-Xss4g" but still no luck..It is
>>>>>>>>> the same stackoverflow error.
>>>>>>>>>
>>>>>>>>> On Fri, Dec 16, 2022 at 9:42 PM Sean Owen <sr...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> OK that's good. Hm, I seem to recall the build needs more mem in
>>>>>>>>>> Java 11 and/or some envs. As a quick check, try replacing all "-Xss4m" with
>>>>>>>>>> "-Xss16m" or something larger, in the project build files. Just search and
>>>>>>>>>> replace.
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 16, 2022 at 9:53 AM Gnana Kumar <
>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I have been following below steps.
>>>>>>>>>>>
>>>>>>>>>>> git clone --branch branch-3.3
>>>>>>>>>>> https://github.com/apache/spark.git
>>>>>>>>>>> cd spark
>>>>>>>>>>> ./dev/make-distribution.sh --tgz --name with-volcano
>>>>>>>>>>> -Pkubernetes,volcano,hadoop-3
>>>>>>>>>>>
>>>>>>>>>>> How to increase stack size ? Please let me know.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 16, 2022 at 8:45 PM Sean Owen <sr...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> You need to increase the stack size during compilation. The
>>>>>>>>>>>> included mvn wrapper in build does this. Are you using it?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 16, 2022 at 9:13 AM Gnana Kumar <
>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> This is my latest error and fails to build SPARK CATALYST
>>>>>>>>>>>>>
>>>>>>>>>>>>> Exception in thread "main" java.lang.StackOverflowError
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1402)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Dec 16, 2022 at 7:16 PM Gnana Kumar <
>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any updates on this please ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Dec 10, 2022 at 1:36 PM Gnana Kumar <
>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Bjorn.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have created Google Cloud Computre Engine and tried
>>>>>>>>>>>>>>> multiple times but getting below error.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have installed JAVA 11 in my VM machine and is there any
>>>>>>>>>>>>>>> other dependency which needs to be installed in VM Machine ? Please let me
>>>>>>>>>>>>>>> know.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1430)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1409)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bjørn Jørgensen
>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>> Norge
>>>>>>>>
>>>>>>>> +47 480 94 297
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Gnana
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks
>>>>>> Gnana
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Gnana
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Gnana
>>>>
>>>
>>>
>>> --
>>> Thanks
>>> Gnana
>>>
>>
>>
>> --
>> Thanks
>> Gnana
>>
>
>
> --
> Thanks
> Gnana
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Bjørn Jørgensen <bj...@gmail.com>.
I use java 17 to build this.
Are there any reasons why you have to build spark yourself? Can't you start
from spark 3.3.1 tar file and build a docker image from there?
fre. 16. des. 2022 kl. 18:26 skrev Gnana Kumar <gn...@gmail.com>:
> I have opened the pom file under cd ~/spark/sql/catalyst$ vi pom.xml and
> increases from "-Xss4m" to "-Xss4g" but still no luck..It is the same
> stackoverflow error.
>
> On Fri, Dec 16, 2022 at 9:42 PM Sean Owen <sr...@gmail.com> wrote:
>
>> OK that's good. Hm, I seem to recall the build needs more mem in Java 11
>> and/or some envs. As a quick check, try replacing all "-Xss4m" with
>> "-Xss16m" or something larger, in the project build files. Just search and
>> replace.
>>
>> On Fri, Dec 16, 2022 at 9:53 AM Gnana Kumar <gn...@gmail.com>
>> wrote:
>>
>>> I have been following below steps.
>>>
>>> git clone --branch branch-3.3 https://github.com/apache/spark.git
>>> cd spark
>>> ./dev/make-distribution.sh --tgz --name with-volcano
>>> -Pkubernetes,volcano,hadoop-3
>>>
>>> How to increase stack size ? Please let me know.
>>>
>>> Thanks
>>> Gnana
>>>
>>> On Fri, Dec 16, 2022 at 8:45 PM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> You need to increase the stack size during compilation. The included
>>>> mvn wrapper in build does this. Are you using it?
>>>>
>>>> On Fri, Dec 16, 2022 at 9:13 AM Gnana Kumar <gn...@gmail.com>
>>>> wrote:
>>>>
>>>>> This is my latest error and fails to build SPARK CATALYST
>>>>>
>>>>> Exception in thread "main" java.lang.StackOverflowError
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1402)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>
>>>>> On Fri, Dec 16, 2022 at 7:16 PM Gnana Kumar <gn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Any updates on this please ?
>>>>>>
>>>>>> On Sat, Dec 10, 2022 at 1:36 PM Gnana Kumar <gn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Bjorn.
>>>>>>>
>>>>>>> I have created Google Cloud Computre Engine and tried multiple times
>>>>>>> but getting below error.
>>>>>>>
>>>>>>> I have installed JAVA 11 in my VM machine and is there any other
>>>>>>> dependency which needs to be installed in VM Machine ? Please let me know.
>>>>>>>
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1430)
>>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
>>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>>> at
>>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>>> at
>>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>>>
>>>>>>>
>>>
>>> --
>>> Thanks
>>> Gnana
>>>
>>
>
> --
> Thanks
> Gnana
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Gnana Kumar <gn...@gmail.com>.
I have opened the pom file under cd ~/spark/sql/catalyst$ vi pom.xml and
increases from "-Xss4m" to "-Xss4g" but still no luck..It is the same
stackoverflow error.
On Fri, Dec 16, 2022 at 9:42 PM Sean Owen <sr...@gmail.com> wrote:
> OK that's good. Hm, I seem to recall the build needs more mem in Java 11
> and/or some envs. As a quick check, try replacing all "-Xss4m" with
> "-Xss16m" or something larger, in the project build files. Just search and
> replace.
>
> On Fri, Dec 16, 2022 at 9:53 AM Gnana Kumar <gn...@gmail.com>
> wrote:
>
>> I have been following below steps.
>>
>> git clone --branch branch-3.3 https://github.com/apache/spark.git
>> cd spark
>> ./dev/make-distribution.sh --tgz --name with-volcano
>> -Pkubernetes,volcano,hadoop-3
>>
>> How to increase stack size ? Please let me know.
>>
>> Thanks
>> Gnana
>>
>> On Fri, Dec 16, 2022 at 8:45 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> You need to increase the stack size during compilation. The included mvn
>>> wrapper in build does this. Are you using it?
>>>
>>> On Fri, Dec 16, 2022 at 9:13 AM Gnana Kumar <gn...@gmail.com>
>>> wrote:
>>>
>>>> This is my latest error and fails to build SPARK CATALYST
>>>>
>>>> Exception in thread "main" java.lang.StackOverflowError
>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1402)
>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>> at
>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>> at
>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>> at
>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>> at
>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>> at
>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>> at
>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>> at
>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>> at
>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>> at
>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>
>>>> On Fri, Dec 16, 2022 at 7:16 PM Gnana Kumar <gn...@gmail.com>
>>>> wrote:
>>>>
>>>>> Any updates on this please ?
>>>>>
>>>>> On Sat, Dec 10, 2022 at 1:36 PM Gnana Kumar <gn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Bjorn.
>>>>>>
>>>>>> I have created Google Cloud Computre Engine and tried multiple times
>>>>>> but getting below error.
>>>>>>
>>>>>> I have installed JAVA 11 in my VM machine and is there any other
>>>>>> dependency which needs to be installed in VM Machine ? Please let me know.
>>>>>>
>>>>>> at
>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>> at
>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>> at
>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1430)
>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>> at
>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>> at
>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>>> at
>>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>>> at
>>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>>> at
>>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>>
>>>>>>
>>
>> --
>> Thanks
>> Gnana
>>
>
--
Thanks
Gnana
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Sean Owen <sr...@gmail.com>.
OK that's good. Hm, I seem to recall the build needs more mem in Java 11
and/or some envs. As a quick check, try replacing all "-Xss4m" with
"-Xss16m" or something larger, in the project build files. Just search and
replace.
On Fri, Dec 16, 2022 at 9:53 AM Gnana Kumar <gn...@gmail.com>
wrote:
> I have been following below steps.
>
> git clone --branch branch-3.3 https://github.com/apache/spark.git
> cd spark
> ./dev/make-distribution.sh --tgz --name with-volcano
> -Pkubernetes,volcano,hadoop-3
>
> How to increase stack size ? Please let me know.
>
> Thanks
> Gnana
>
> On Fri, Dec 16, 2022 at 8:45 PM Sean Owen <sr...@gmail.com> wrote:
>
>> You need to increase the stack size during compilation. The included mvn
>> wrapper in build does this. Are you using it?
>>
>> On Fri, Dec 16, 2022 at 9:13 AM Gnana Kumar <gn...@gmail.com>
>> wrote:
>>
>>> This is my latest error and fails to build SPARK CATALYST
>>>
>>> Exception in thread "main" java.lang.StackOverflowError
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1402)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>
>>> On Fri, Dec 16, 2022 at 7:16 PM Gnana Kumar <gn...@gmail.com>
>>> wrote:
>>>
>>>> Any updates on this please ?
>>>>
>>>> On Sat, Dec 10, 2022 at 1:36 PM Gnana Kumar <gn...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Bjorn.
>>>>>
>>>>> I have created Google Cloud Computre Engine and tried multiple times
>>>>> but getting below error.
>>>>>
>>>>> I have installed JAVA 11 in my VM machine and is there any other
>>>>> dependency which needs to be installed in VM Machine ? Please let me know.
>>>>>
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1430)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>>>> at
>>>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>>>> at
>>>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>>>> at
>>>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>>>
>>>>>
>
> --
> Thanks
> Gnana
>
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Sean Owen <sr...@gmail.com>.
You need to increase the stack size during compilation. The included mvn
wrapper in build does this. Are you using it?
On Fri, Dec 16, 2022 at 9:13 AM Gnana Kumar <gn...@gmail.com>
wrote:
> This is my latest error and fails to build SPARK CATALYST
>
> Exception in thread "main" java.lang.StackOverflowError
> at scala.reflect.internal.Trees.itransform(Trees.scala:1402)
> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
> at
> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
> at
> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
> at
> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
> at
> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
> at
> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
> at
> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
> at
> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>
> On Fri, Dec 16, 2022 at 7:16 PM Gnana Kumar <gn...@gmail.com>
> wrote:
>
>> Any updates on this please ?
>>
>> On Sat, Dec 10, 2022 at 1:36 PM Gnana Kumar <gn...@gmail.com>
>> wrote:
>>
>>> Thanks Bjorn.
>>>
>>> I have created Google Cloud Computre Engine and tried multiple times but
>>> getting below error.
>>>
>>> I have installed JAVA 11 in my VM machine and is there any other
>>> dependency which needs to be installed in VM Machine ? Please let me know.
>>>
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1430)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>>> at
>>> scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>>>
>>>
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Gnana Kumar <gn...@gmail.com>.
This is my latest error and fails to build SPARK CATALYST
Exception in thread "main" java.lang.StackOverflowError
at scala.reflect.internal.Trees.itransform(Trees.scala:1402)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
On Fri, Dec 16, 2022 at 7:16 PM Gnana Kumar <gn...@gmail.com>
wrote:
> Any updates on this please ?
>
> On Sat, Dec 10, 2022 at 1:36 PM Gnana Kumar <gn...@gmail.com>
> wrote:
>
>> Thanks Bjorn.
>>
>> I have created Google Cloud Computre Engine and tried multiple times but
>> getting below error.
>>
>> I have installed JAVA 11 in my VM machine and is there any other
>> dependency which needs to be installed in VM Machine ? Please let me know.
>>
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> at scala.reflect.internal.Trees.itransform(Trees.scala:1430)
>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
>> at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at
>> scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
>> at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
>> gnana_kumar123@instance-1:~/spark$
>> g
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary for Spark Project Parent POM 3.3.2-SNAPSHOT:
>> [INFO]
>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>> 4.708 s]
>> [INFO] Spark Project Tags ................................. SUCCESS [
>> 11.329 s]
>> [INFO] Spark Project Sketch ............................... SUCCESS [
>> 11.701 s]
>> [INFO] Spark Project Local DB ............................. SUCCESS [
>> 8.626 s]
>> [INFO] Spark Project Networking ........................... SUCCESS [
>> 13.244 s]
>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>> 10.251 s]
>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>> 16.433 s]
>> [INFO] Spark Project Launcher ............................. SUCCESS [
>> 7.779 s]
>> [INFO] Spark Project Core ................................. SUCCESS
>> [04:42 min]
>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>> 42.330 s]
>> [INFO] Spark Project GraphX ............................... SUCCESS [
>> 49.378 s]
>> [INFO] Spark Project Streaming ............................ SUCCESS
>> [01:27 min]
>> [INFO] Spark Project Catalyst ............................. FAILURE
>> [01:08 min]
>> [INFO] Spark Project SQL .................................. SKIPPED
>> [INFO] Spark Project ML Library ........................... SKIPPED
>> [INFO] Spark Project Tools ................................ SKIPPED
>> [INFO] Spark Project Hive ................................. SKIPPED
>> [INFO] Spark Project REPL ................................. SKIPPED
>> [INFO] Spark Project Kubernetes ........................... SKIPPED
>> [INFO] Spark Project Assembly ............................. SKIPPED
>> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
>> [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
>> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
>> [INFO] Spark Project Examples ............................. SKIPPED
>>
>> On Thu, Dec 8, 2022 at 11:11 PM Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> test_volcano]$ git clone --branch branch-3.3
>>> https://github.com/apache/spark.git
>>> Cloning into 'spark'...
>>> remote: Enumerating objects: 959089, done.
>>> remote: Counting objects: 100% (355/355), done.
>>> remote: Compressing objects: 100% (220/220), done.
>>> remote: Total 959089 (delta 102), reused 235 (delta 55), pack-reused
>>> 958734
>>> Receiving objects: 100% (959089/959089), 438.07 MiB | 19.42 MiB/s, done.
>>> Resolving deltas: 100% (426020/426020), done.
>>> test_volcano]$ cd spark/
>>>
>>> ./dev/make-distribution.sh --tgz --name with-volcano
>>> -Pkubernetes,volcano,hadoop-3
>>>
>>> (...)
>>>
>>> [INFO] Reactor Summary for Spark Project Parent POM 3.3.2-SNAPSHOT:
>>> [INFO]
>>> [INFO] Spark Project Parent POM ........................... SUCCESS
>>> [01:27 min]
>>> [INFO] Spark Project Tags ................................. SUCCESS [
>>> 17.137 s]
>>> [INFO] Spark Project Sketch ............................... SUCCESS [
>>> 4.227 s]
>>> [INFO] Spark Project Local DB ............................. SUCCESS [
>>> 12.887 s]
>>> [INFO] Spark Project Networking ........................... SUCCESS [
>>> 22.413 s]
>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>>> 3.734 s]
>>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>>> 6.212 s]
>>> [INFO] Spark Project Launcher ............................. SUCCESS [
>>> 7.791 s]
>>> [INFO] Spark Project Core ................................. SUCCESS
>>> [01:52 min]
>>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>>> 21.201 s]
>>> [INFO] Spark Project GraphX ............................... SUCCESS [
>>> 20.063 s]
>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>> 34.545 s]
>>> [INFO] Spark Project Catalyst ............................. SUCCESS
>>> [01:40 min]
>>> [INFO] Spark Project SQL .................................. SUCCESS
>>> [02:47 min]
>>> [INFO] Spark Project ML Library ........................... SUCCESS
>>> [01:28 min]
>>> [INFO] Spark Project Tools ................................ SUCCESS [
>>> 3.459 s]
>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>> 52.119 s]
>>> [INFO] Spark Project REPL ................................. SUCCESS [
>>> 11.587 s]
>>> [INFO] Spark Project Kubernetes ........................... SUCCESS
>>> [01:03 min]
>>> [INFO] Spark Project Assembly ............................. SUCCESS [
>>> 1.484 s]
>>> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
>>> 11.976 s]
>>> [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [
>>> 18.443 s]
>>> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [
>>> 29.297 s]
>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>> 22.312 s]
>>> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
>>> 6.321 s]
>>> [INFO] Spark Avro ......................................... SUCCESS [
>>> 25.435 s]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 15:53 min
>>> [INFO] Finished at: 2022-12-08T18:32:50+01:00
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> + rm -rf /home/bjorn/test_volcano/spark/dist
>>> + mkdir -p /home/bjorn/test_volcano/spark/dist/jars
>>> + echo 'Spark 3.3.2-SNAPSHOT (git revision 02f32ee358) built for Hadoop
>>> 3.3.2'
>>> + echo 'Build flags: -Pkubernetes,volcano,hadoop-3'
>>> + cp
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/activation-1.1.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/aircompressor-0.21.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/algebra_2.12-2.0.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/annotations-17.0.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/antlr4-runtime-4.8.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/aopalliance-repackaged-2.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arpack-2.2.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arpack_combined_all-0.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-format-7.0.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-memory-core-7.0.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-memory-netty-7.0.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-vector-7.0.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/audience-annotations-0.5.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/automaton-1.11-8.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-1.11.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-ipc-1.11.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-mapred-1.11.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/blas-2.2.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/breeze_2.12-1.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/breeze-macros_2.12-1.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/cats-kernel_2.12-2.1.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/chill_2.12-0.10.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/chill-java-0.10.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-codec-1.15.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-collections-3.2.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-collections4-4.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-compiler-3.0.16.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-compress-1.21.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-crypto-1.1.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-io-2.11.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-lang-2.6.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-lang3-3.12.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-logging-1.1.3.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-math3-3.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-text-1.10.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/compress-lzf-1.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/core-1.1.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-client-2.13.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-framework-2.13.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-recipes-2.13.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/flatbuffers-java-1.12.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/generex-1.0.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/gson-2.8.6.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/guava-14.0.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hadoop-client-api-3.3.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hadoop-client-runtime-3.3.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hive-storage-api-2.7.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-api-2.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-locator-2.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-utils-2.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/istack-commons-runtime-3.0.8.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/ivy-2.5.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-annotations-2.13.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-core-2.13.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-databind-2.13.4.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-dataformat-yaml-2.13.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-datatype-jsr310-2.13.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-module-scala_2.12-2.13.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.annotation-api-1.3.5.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.inject-2.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.servlet-api-4.0.3.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.validation-api-2.0.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.ws.rs-api-2.1.6.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.xml.bind-api-2.3.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/janino-3.0.16.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/javassist-3.25.0-GA.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jaxb-runtime-2.3.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jcl-over-slf4j-1.7.32.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-client-2.36.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-common-2.36.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-container-servlet-2.36.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-container-servlet-core-2.36.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-hk2-2.36.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-server-2.36.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/JLargeArrays-1.5.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-ast_2.12-3.7.0-M11.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-core_2.12-3.7.0-M11.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-jackson_2.12-3.7.0-M11.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-scalap_2.12-3.7.0-M11.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jsr305-3.0.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/JTransforms-3.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jul-to-slf4j-1.7.32.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kryo-shaded-4.0.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-client-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-admissionregistration-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-apiextensions-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-apps-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-autoscaling-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-batch-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-certificates-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-common-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-coordination-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-core-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-discovery-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-events-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-extensions-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-flowcontrol-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-metrics-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-networking-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-node-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-policy-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-rbac-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-scheduling-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-storageclass-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/lapack-2.2.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/leveldbjni-all-1.8.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-1.2-api-2.17.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-api-2.17.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-core-2.17.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-slf4j-impl-2.17.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/logging-interceptor-3.12.12.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/lz4-java-1.8.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-core-4.2.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-graphite-4.2.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-jmx-4.2.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-json-4.2.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-jvm-4.2.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/minlog-1.3.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-all-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-buffer-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-codec-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-common-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-handler-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-resolver-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-tcnative-classes-2.0.48.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-classes-epoll-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-classes-kqueue-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-epoll-4.1.74.Final-linux-aarch_64.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-epoll-4.1.74.Final-linux-x86_64.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-kqueue-4.1.74.Final-osx-aarch_64.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-kqueue-4.1.74.Final-osx-x86_64.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-unix-common-4.1.74.Final.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/objenesis-3.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/okhttp-3.12.12.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/okio-1.14.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/opencsv-2.3.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-core-1.7.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-mapreduce-1.7.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-shims-1.7.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/oro-2.0.8.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/osgi-resource-locator-1.0.3.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/paranamer-2.8.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-column-1.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-common-1.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-encoding-1.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-format-structures-1.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-hadoop-1.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-jackson-1.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/pickle-1.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/protobuf-java-2.5.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/py4j-0.10.9.5.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/RoaringBitmap-0.9.25.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/rocksdbjni-6.20.3.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-collection-compat_2.12-2.1.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-compiler-2.12.15.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-library-2.12.15.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-parser-combinators_2.12-1.1.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-reflect-2.12.15.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-xml_2.12-1.2.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/shapeless_2.12-2.3.7.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/shims-0.9.25.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/slf4j-api-1.7.32.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/snakeyaml-1.31.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/snappy-java-1.1.8.4.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-catalyst_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-core_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-graphx_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-kvstore_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-launcher_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-mllib_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-mllib-local_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-network-common_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-network-shuffle_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-repl_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-sketch_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-sql_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-streaming_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-tags_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-tags_2.12-3.3.2-SNAPSHOT-tests.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-unsafe_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire_2.12-0.17.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-macros_2.12-0.17.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-platform_2.12-0.17.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-util_2.12-0.17.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/stream-2.9.6.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/threeten-extra-1.5.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/tink-1.6.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/univocity-parsers-2.9.1.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/volcano-client-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/volcano-model-v1beta1-5.12.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/xbean-asm9-shaded-4.20.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/xz-1.9.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zjsonpatch-0.3.0.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zookeeper-3.6.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zookeeper-jute-3.6.2.jar
>>> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zstd-jni-1.5.2-1.jar
>>> /home/bjorn/test_volcano/spark/dist/jars/
>>> + '[' -f
>>> '/home/bjorn/test_volcano/spark/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar'
>>> ']'
>>> + '[' -d
>>> /home/bjorn/test_volcano/spark/resource-managers/kubernetes/core/target/ ']'
>>> + mkdir -p /home/bjorn/test_volcano/spark/dist/kubernetes/
>>> + cp -a
>>> /home/bjorn/test_volcano/spark/resource-managers/kubernetes/docker/src/main/dockerfiles
>>> /home/bjorn/test_volcano/spark/dist/kubernetes/
>>> + cp -a
>>> /home/bjorn/test_volcano/spark/resource-managers/kubernetes/integration-tests/tests
>>> /home/bjorn/test_volcano/spark/dist/kubernetes/
>>> + mkdir -p /home/bjorn/test_volcano/spark/dist/examples/jars
>>> + cp
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/aircompressor-0.21.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/annotations-17.0.0.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/commons-codec-1.15.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/commons-lang3-3.12.0.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/hive-storage-api-2.7.2.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/istack-commons-runtime-3.0.8.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/jakarta.xml.bind-api-2.3.2.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/jaxb-runtime-2.3.2.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-core-1.7.7.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-mapreduce-1.7.7.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-shims-1.7.7.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/scopt_2.12-3.7.1.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
>>> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/threeten-extra-1.5.0.jar
>>> /home/bjorn/test_volcano/spark/dist/examples/jars
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/aircompressor-0.21.jar
>>> + name=aircompressor-0.21.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/aircompressor-0.21.jar
>>> ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/aircompressor-0.21.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/annotations-17.0.0.jar
>>> + name=annotations-17.0.0.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/annotations-17.0.0.jar
>>> ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/annotations-17.0.0.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-codec-1.15.jar
>>> + name=commons-codec-1.15.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/commons-codec-1.15.jar
>>> ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-codec-1.15.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-lang3-3.12.0.jar
>>> + name=commons-lang3-3.12.0.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/commons-lang3-3.12.0.jar ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-lang3-3.12.0.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/hive-storage-api-2.7.2.jar
>>> + name=hive-storage-api-2.7.2.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/hive-storage-api-2.7.2.jar ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/hive-storage-api-2.7.2.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/istack-commons-runtime-3.0.8.jar
>>> + name=istack-commons-runtime-3.0.8.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/istack-commons-runtime-3.0.8.jar
>>> ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/istack-commons-runtime-3.0.8.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/jakarta.xml.bind-api-2.3.2.jar
>>> + name=jakarta.xml.bind-api-2.3.2.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/jakarta.xml.bind-api-2.3.2.jar ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/jakarta.xml.bind-api-2.3.2.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/jaxb-runtime-2.3.2.jar
>>> + name=jaxb-runtime-2.3.2.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/jaxb-runtime-2.3.2.jar
>>> ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/jaxb-runtime-2.3.2.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-core-1.7.7.jar
>>> + name=orc-core-1.7.7.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-core-1.7.7.jar ']'
>>> + rm /home/bjorn/test_volcano/spark/dist/examples/jars/orc-core-1.7.7.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-mapreduce-1.7.7.jar
>>> + name=orc-mapreduce-1.7.7.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/orc-mapreduce-1.7.7.jar ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-mapreduce-1.7.7.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-shims-1.7.7.jar
>>> + name=orc-shims-1.7.7.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-shims-1.7.7.jar ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-shims-1.7.7.jar
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/scopt_2.12-3.7.1.jar
>>> + name=scopt_2.12-3.7.1.jar
>>> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/scopt_2.12-3.7.1.jar
>>> ']'
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
>>> + name=spark-examples_2.12-3.3.2-SNAPSHOT.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
>>> ']'
>>> + for f in "$DISTDIR"/examples/jars/*
>>> ++ basename
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/threeten-extra-1.5.0.jar
>>> + name=threeten-extra-1.5.0.jar
>>> + '[' -f
>>> /home/bjorn/test_volcano/spark/dist/jars/threeten-extra-1.5.0.jar ']'
>>> + rm
>>> /home/bjorn/test_volcano/spark/dist/examples/jars/threeten-extra-1.5.0.jar
>>> + mkdir -p /home/bjorn/test_volcano/spark/dist/examples/src/main
>>> + cp -r /home/bjorn/test_volcano/spark/examples/src/main
>>> /home/bjorn/test_volcano/spark/dist/examples/src/
>>> + '[' -e /home/bjorn/test_volcano/spark/LICENSE-binary ']'
>>> + cp /home/bjorn/test_volcano/spark/LICENSE-binary
>>> /home/bjorn/test_volcano/spark/dist/LICENSE
>>> + cp -r /home/bjorn/test_volcano/spark/licenses-binary
>>> /home/bjorn/test_volcano/spark/dist/licenses
>>> + cp /home/bjorn/test_volcano/spark/NOTICE-binary
>>> /home/bjorn/test_volcano/spark/dist/NOTICE
>>> + '[' -e /home/bjorn/test_volcano/spark/CHANGES.txt ']'
>>> + cp -r /home/bjorn/test_volcano/spark/data
>>> /home/bjorn/test_volcano/spark/dist
>>> + '[' false == true ']'
>>> + echo 'Skipping building python distribution package'
>>> Skipping building python distribution package
>>> + '[' false == true ']'
>>> + echo 'Skipping building R source package'
>>> Skipping building R source package
>>> + mkdir /home/bjorn/test_volcano/spark/dist/conf
>>> + cp /home/bjorn/test_volcano/spark/conf/fairscheduler.xml.template
>>> /home/bjorn/test_volcano/spark/conf/log4j2.properties.template
>>> /home/bjorn/test_volcano/spark/conf/metrics.properties.template
>>> /home/bjorn/test_volcano/spark/conf/spark-defaults.conf.template
>>> /home/bjorn/test_volcano/spark/conf/spark-env.sh.template
>>> /home/bjorn/test_volcano/spark/conf/workers.template
>>> /home/bjorn/test_volcano/spark/dist/conf
>>> + cp /home/bjorn/test_volcano/spark/README.md
>>> /home/bjorn/test_volcano/spark/dist
>>> + cp -r /home/bjorn/test_volcano/spark/bin
>>> /home/bjorn/test_volcano/spark/dist
>>> + cp -r /home/bjorn/test_volcano/spark/python
>>> /home/bjorn/test_volcano/spark/dist
>>> + '[' false == true ']'
>>> + cp -r /home/bjorn/test_volcano/spark/sbin
>>> /home/bjorn/test_volcano/spark/dist
>>> + '[' -d /home/bjorn/test_volcano/spark/R/lib/SparkR ']'
>>> + '[' true == true ']'
>>> + TARDIR_NAME=spark-3.3.2-SNAPSHOT-bin-with-volcano
>>> +
>>> TARDIR=/home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
>>> + rm -rf
>>> /home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
>>> + cp -r /home/bjorn/test_volcano/spark/dist
>>> /home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
>>> + tar czf spark-3.3.2-SNAPSHOT-bin-with-volcano.tgz -C
>>> /home/bjorn/test_volcano/spark spark-3.3.2-SNAPSHOT-bin-with-volcano
>>> + rm -rf
>>> /home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
>>>
>>>
>>>
>>> spark]$ ls
>>> appveyor.yml bin build conf core dev docs
>>> external hadoop-cloud LICENSE licenses mllib
>>> NOTICE pom.xml python README.md resource-managers
>>> scalastyle-config.xml sql target
>>> assembly binder common CONTRIBUTING.md data dist examples
>>> graphx launcher LICENSE-binary licenses-binary mllib-local
>>> NOTICE-binary project R repl sbin
>>> spark-3.3.2-SNAPSHOT-bin-with-volcano.tgz streaming tools
>>>
>>> tor. 8. des. 2022 kl. 17:31 skrev Gnana Kumar <gnana.kumar123@gmail.com
>>> >:
>>>
>>>> Hi There,
>>>>
>>>> I have been trying to follow the steps as per above url but below
>>>> command does not successfully create the binary distribution of spark with
>>>> volcano.
>>>>
>>>> ./dev/make-distribution.sh --tgz --name with-volcano
>>>> -Pkubernetes,volcano,hadoop-3
>>>>
>>>> Has anyone tried it and it works ? please confirm.
>>>>
>>>> Thanks
>>>> Gnana
>>>>
>>>> On Tue, Nov 29, 2022 at 10:16 PM Bjørn Jørgensen <
>>>> bjornjorgensen@gmail.com> wrote:
>>>>
>>>>> MartinG have wrote a good howto on this
>>>>>
>>>>>
>>>>> https://martin-grigorov.medium.com/apache-spark-snapshot-on-kubernetes-178592613408
>>>>>
>>>>>
>>>>> https://martin-grigorov.medium.com/native-integration-between-apache-spark-and-volcano-kubernetes-scheduler-488f54dbbab3
>>>>>
>>>>>
>>>>> tir. 29. nov. 2022 kl. 16:25 skrev Gnana Kumar <
>>>>> gnana.kumar123@gmail.com>:
>>>>>
>>>>>> Any updates on this please ?
>>>>>>
>>>>>> On Mon, Nov 28, 2022 at 11:58 AM Gnana Kumar <
>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, while building the binary dist, I have used the -Pvolcano
>>>>>>> option but for the docker build , do we have any options to use ?
>>>>>>>
>>>>>>> >docker build -t gnana_volcano_scheduler_snapshot -f
>>>>>>> kubernetes/dockerfiles/spark/Dockerfile .
>>>>>>>
>>>>>>> Because my understanding is that the binary dist will have all jars
>>>>>>> which should be packed within the container.
>>>>>>>
>>>>>>> Please let us know how to build docker image with volcano options.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Gnana
>>>>>>>
>>>>>>> On Sun, Nov 27, 2022 at 3:41 PM Denis Bolshakov <
>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>
>>>>>>>> I am talking about docker image, not about your local build. Could
>>>>>>>> you please make sure that docker image has a spark distro with volcano
>>>>>>>> support?
>>>>>>>>
>>>>>>>> вс, 27 нояб. 2022 г., 13:09 Gnana Kumar <gn...@gmail.com>:
>>>>>>>>
>>>>>>>>> Yes. I have cloned the spark source 3.3.2.SNAPSHOT and then used
>>>>>>>>> below command to build the binary distribution with support of volcano.
>>>>>>>>>
>>>>>>>>> >dev/./make-distribution.sh -Denforcer.skip=true
>>>>>>>>> -DrecompileMode=all -Pkubernetes -Pvolcano -Pscala-2.12 -DskipTests
>>>>>>>>>
>>>>>>>>> Also please find the attached screenshot of spark-kubernetes JAR
>>>>>>>>> built from above command and if we unzip,it has the scala class -
>>>>>>>>> org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala
>>>>>>>>>
>>>>>>>>> [image: image.png]
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>> On Sun, Nov 27, 2022 at 3:23 PM Denis Bolshakov <
>>>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Does your driver docker image has a spark distro built with
>>>>>>>>>> volcano support?
>>>>>>>>>>
>>>>>>>>>> вс, 27 нояб. 2022 г., 11:55 Gnana Kumar <gnana.kumar123@gmail.com
>>>>>>>>>> >:
>>>>>>>>>>
>>>>>>>>>>> Actual error content is pasted below.
>>>>>>>>>>>
>>>>>>>>>>> Exception in thread "main"* java.lang.ClassNotFoundException:
>>>>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep*
>>>>>>>>>>> at
>>>>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>>>>> at
>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>>>>> at
>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>>>>>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.TraversableLike.map(TraversableLike.scala:286)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.TraversableLike.map$(TraversableLike.scala:279)
>>>>>>>>>>> at
>>>>>>>>>>> scala.collection.AbstractTraversable.map(Traversable.scala:108)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:41)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:106)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:220)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:214)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2713)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:214)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:186)
>>>>>>>>>>> at org.apache.spark.deploy.SparkSubmit.org
>>>>>>>>>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>>>>>>> 22/11/27 08:35:12 INFO ShutdownHookManager: Shutdown hook called
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 27, 2022 at 2:11 PM Gnana Kumar <
>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Though my
>>>>>>>>>>>> /opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar has
>>>>>>>>>>>> VolcanoFeatureStep.class sparksubmit says class is not found.
>>>>>>>>>>>>
>>>>>>>>>>>> Basically I'm looking for a working example of spark submit
>>>>>>>>>>>> with Volcano scheduler in Kubernetes.
>>>>>>>>>>>>
>>>>>>>>>>>> Please help me resolve this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> ~/spark_3.3.2/bin/spark-submit \
>>>>>>>>>>>> --verbose \
>>>>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>>>>>> --deploy-mode cluster \
>>>>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.driver.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.executor.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>>>>> --conf spark.kubernetes.file.upload.path=file:///tmp
>>>>>>>>>>>> \
>>>>>>>>>>>> --jars
>>>>>>>>>>>> local:///opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar \
>>>>>>>>>>>>
>>>>>>>>>>>> local:///opt/spark/examples/jars/spark-analytics-application-1.0.jar
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> GK
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Nov 27, 2022 at 2:03 PM Gnana Kumar <
>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, I have installed it using the command below and volcano
>>>>>>>>>>>>> containers are coming up successfully in my GKE cluster.
>>>>>>>>>>>>>
>>>>>>>>>>>>> >kubectl apply -f
>>>>>>>>>>>>> https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/installer/volcano-development.yaml
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>>
>>>>>>>>>>>>> However Spark submit cannot refer to the *VolcanoFeatureSetp
>>>>>>>>>>>>> class *and it says class has not found exception.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Nov 27, 2022 at 2:53 AM Bjørn Jørgensen <
>>>>>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have you installed volcano to the k8s cluster?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> https://volcano.sh/en/docs/spark_on_volcano/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> lør. 26. nov. 2022 kl. 09:06 skrev Gnana Kumar <
>>>>>>>>>>>>>> gnana.kumar123@gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have also attached jar which has VolcanoFeature.class.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But when I run from SparkSubmit, it says class not found.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # This is required for volcano
>>>>>>>>>>>>>>> ~/temp/spark_3.3.2/bin/spark-submit \
>>>>>>>>>>>>>>> --verbose \
>>>>>>>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443
>>>>>>>>>>>>>>> \
>>>>>>>>>>>>>>> --deploy-mode cluster \
>>>>>>>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>>>>>>>> \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>> \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>> \
>>>>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>>>>>>>> \
>>>>>>>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please help me on how to run volcano using Spark Submit.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Nov 26, 2022 at 11:43 AM Gnana Kumar <
>>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HI Denis,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is how I have built it from the parent 'spark' dir.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >mvn -Denforcer.skip=true -DrecompileMode=all -Pkubernetes
>>>>>>>>>>>>>>>> -Pvolcano -Pscala-2.12 -DskipTests clean package
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also once the build is completed, the generated
>>>>>>>>>>>>>>>> spark-kubernetes...jar has VolcanoFeature.class but when this class name is
>>>>>>>>>>>>>>>> referred in Spark Submit ,it gives the class not found exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Nov 25, 2022 at 1:43 PM Denis Bolshakov <
>>>>>>>>>>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think volcano is not enabled in default build, you have
>>>>>>>>>>>>>>>>> to provide a maven profile by adding -Pvolcano
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, 24 Nov 2022 at 19:09, Gnana Kumar <
>>>>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there any issue SNAPSHOT jars ?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Nov 24, 2022 at 9:33 PM Gnana Kumar <
>>>>>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi There,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have cloned spark 3.3.2 snapshot source code where
>>>>>>>>>>>>>>>>>>> Volcano feature is present and then I built the binary distribution for the
>>>>>>>>>>>>>>>>>>> same.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> However when I tried to use this spark 3.3.2 SNAPSHOT
>>>>>>>>>>>>>>>>>>> dist with Volcano but I have got class not found exception.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>>>>>>>>>>>> at java.base/java.lang.Class.forName0(Native
>>>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Any help to resolve this issue please ? Basically I
>>>>>>>>>>>>>>>>>>> would like to use Volcano from Spark 3.3.2 in Kubernetes cluster.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> //with Best Regards
>>>>>>>>>>>>>>>>> --Denis Bolshakov
>>>>>>>>>>>>>>>>> e-mail: bolshakov.denis@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>>>>>> Norge
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +47 480 94 297
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gnana
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Gnana
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks
>>>>>> Gnana
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bjørn Jørgensen
>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>> Norge
>>>>>
>>>>> +47 480 94 297
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Gnana
>>>>
>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>>
>> --
>> Thanks
>> Gnana
>>
>
>
> --
> Thanks
> Gnana
>
--
Thanks
Gnana
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Gnana Kumar <gn...@gmail.com>.
Thanks Bjorn.
I have created Google Cloud Computre Engine and tried multiple times but
getting below error.
I have installed JAVA 11 in my VM machine and is there any other dependency
which needs to be installed in VM Machine ? Please let me know.
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1430)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1409)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1436)
at scala.reflect.internal.Trees.itransform$(Trees.scala:1400)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:57)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
at scala.reflect.internal.Trees.itransform(Trees.scala:1411)
gnana_kumar123@instance-1:~/spark$
g
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.3.2-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [
4.708 s]
[INFO] Spark Project Tags ................................. SUCCESS [
11.329 s]
[INFO] Spark Project Sketch ............................... SUCCESS [
11.701 s]
[INFO] Spark Project Local DB ............................. SUCCESS [
8.626 s]
[INFO] Spark Project Networking ........................... SUCCESS [
13.244 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
10.251 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [
16.433 s]
[INFO] Spark Project Launcher ............................. SUCCESS [
7.779 s]
[INFO] Spark Project Core ................................. SUCCESS [04:42
min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [
42.330 s]
[INFO] Spark Project GraphX ............................... SUCCESS [
49.378 s]
[INFO] Spark Project Streaming ............................ SUCCESS [01:27
min]
[INFO] Spark Project Catalyst ............................. FAILURE [01:08
min]
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
On Thu, Dec 8, 2022 at 11:11 PM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> test_volcano]$ git clone --branch branch-3.3
> https://github.com/apache/spark.git
> Cloning into 'spark'...
> remote: Enumerating objects: 959089, done.
> remote: Counting objects: 100% (355/355), done.
> remote: Compressing objects: 100% (220/220), done.
> remote: Total 959089 (delta 102), reused 235 (delta 55), pack-reused 958734
> Receiving objects: 100% (959089/959089), 438.07 MiB | 19.42 MiB/s, done.
> Resolving deltas: 100% (426020/426020), done.
> test_volcano]$ cd spark/
>
> ./dev/make-distribution.sh --tgz --name with-volcano
> -Pkubernetes,volcano,hadoop-3
>
> (...)
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.3.2-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [01:27
> min]
> [INFO] Spark Project Tags ................................. SUCCESS [
> 17.137 s]
> [INFO] Spark Project Sketch ............................... SUCCESS [
> 4.227 s]
> [INFO] Spark Project Local DB ............................. SUCCESS [
> 12.887 s]
> [INFO] Spark Project Networking ........................... SUCCESS [
> 22.413 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
> 3.734 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [
> 6.212 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [
> 7.791 s]
> [INFO] Spark Project Core ................................. SUCCESS [01:52
> min]
> [INFO] Spark Project ML Local Library ..................... SUCCESS [
> 21.201 s]
> [INFO] Spark Project GraphX ............................... SUCCESS [
> 20.063 s]
> [INFO] Spark Project Streaming ............................ SUCCESS [
> 34.545 s]
> [INFO] Spark Project Catalyst ............................. SUCCESS [01:40
> min]
> [INFO] Spark Project SQL .................................. SUCCESS [02:47
> min]
> [INFO] Spark Project ML Library ........................... SUCCESS [01:28
> min]
> [INFO] Spark Project Tools ................................ SUCCESS [
> 3.459 s]
> [INFO] Spark Project Hive ................................. SUCCESS [
> 52.119 s]
> [INFO] Spark Project REPL ................................. SUCCESS [
> 11.587 s]
> [INFO] Spark Project Kubernetes ........................... SUCCESS [01:03
> min]
> [INFO] Spark Project Assembly ............................. SUCCESS [
> 1.484 s]
> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
> 11.976 s]
> [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [
> 18.443 s]
> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [
> 29.297 s]
> [INFO] Spark Project Examples ............................. SUCCESS [
> 22.312 s]
> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
> 6.321 s]
> [INFO] Spark Avro ......................................... SUCCESS [
> 25.435 s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 15:53 min
> [INFO] Finished at: 2022-12-08T18:32:50+01:00
> [INFO]
> ------------------------------------------------------------------------
> + rm -rf /home/bjorn/test_volcano/spark/dist
> + mkdir -p /home/bjorn/test_volcano/spark/dist/jars
> + echo 'Spark 3.3.2-SNAPSHOT (git revision 02f32ee358) built for Hadoop
> 3.3.2'
> + echo 'Build flags: -Pkubernetes,volcano,hadoop-3'
> + cp
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/activation-1.1.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/aircompressor-0.21.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/algebra_2.12-2.0.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/annotations-17.0.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/antlr4-runtime-4.8.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/aopalliance-repackaged-2.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arpack-2.2.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arpack_combined_all-0.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-format-7.0.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-memory-core-7.0.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-memory-netty-7.0.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-vector-7.0.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/audience-annotations-0.5.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/automaton-1.11-8.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-1.11.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-ipc-1.11.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-mapred-1.11.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/blas-2.2.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/breeze_2.12-1.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/breeze-macros_2.12-1.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/cats-kernel_2.12-2.1.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/chill_2.12-0.10.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/chill-java-0.10.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-codec-1.15.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-collections-3.2.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-collections4-4.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-compiler-3.0.16.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-compress-1.21.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-crypto-1.1.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-io-2.11.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-lang-2.6.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-lang3-3.12.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-logging-1.1.3.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-math3-3.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-text-1.10.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/compress-lzf-1.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/core-1.1.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-client-2.13.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-framework-2.13.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-recipes-2.13.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/flatbuffers-java-1.12.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/generex-1.0.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/gson-2.8.6.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/guava-14.0.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hadoop-client-api-3.3.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hadoop-client-runtime-3.3.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hive-storage-api-2.7.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-api-2.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-locator-2.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-utils-2.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/istack-commons-runtime-3.0.8.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/ivy-2.5.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-annotations-2.13.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-core-2.13.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-databind-2.13.4.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-dataformat-yaml-2.13.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-datatype-jsr310-2.13.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-module-scala_2.12-2.13.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.annotation-api-1.3.5.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.inject-2.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.servlet-api-4.0.3.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.validation-api-2.0.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.ws.rs-api-2.1.6.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.xml.bind-api-2.3.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/janino-3.0.16.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/javassist-3.25.0-GA.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jaxb-runtime-2.3.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jcl-over-slf4j-1.7.32.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-client-2.36.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-common-2.36.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-container-servlet-2.36.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-container-servlet-core-2.36.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-hk2-2.36.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-server-2.36.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/JLargeArrays-1.5.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-ast_2.12-3.7.0-M11.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-core_2.12-3.7.0-M11.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-jackson_2.12-3.7.0-M11.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-scalap_2.12-3.7.0-M11.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jsr305-3.0.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/JTransforms-3.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jul-to-slf4j-1.7.32.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kryo-shaded-4.0.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-client-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-admissionregistration-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-apiextensions-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-apps-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-autoscaling-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-batch-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-certificates-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-common-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-coordination-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-core-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-discovery-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-events-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-extensions-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-flowcontrol-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-metrics-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-networking-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-node-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-policy-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-rbac-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-scheduling-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-storageclass-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/lapack-2.2.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/leveldbjni-all-1.8.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-1.2-api-2.17.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-api-2.17.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-core-2.17.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-slf4j-impl-2.17.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/logging-interceptor-3.12.12.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/lz4-java-1.8.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-core-4.2.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-graphite-4.2.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-jmx-4.2.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-json-4.2.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-jvm-4.2.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/minlog-1.3.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-all-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-buffer-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-codec-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-common-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-handler-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-resolver-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-tcnative-classes-2.0.48.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-classes-epoll-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-classes-kqueue-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-epoll-4.1.74.Final-linux-aarch_64.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-epoll-4.1.74.Final-linux-x86_64.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-kqueue-4.1.74.Final-osx-aarch_64.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-kqueue-4.1.74.Final-osx-x86_64.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-unix-common-4.1.74.Final.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/objenesis-3.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/okhttp-3.12.12.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/okio-1.14.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/opencsv-2.3.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-core-1.7.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-mapreduce-1.7.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-shims-1.7.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/oro-2.0.8.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/osgi-resource-locator-1.0.3.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/paranamer-2.8.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-column-1.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-common-1.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-encoding-1.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-format-structures-1.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-hadoop-1.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-jackson-1.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/pickle-1.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/protobuf-java-2.5.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/py4j-0.10.9.5.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/RoaringBitmap-0.9.25.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/rocksdbjni-6.20.3.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-collection-compat_2.12-2.1.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-compiler-2.12.15.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-library-2.12.15.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-parser-combinators_2.12-1.1.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-reflect-2.12.15.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-xml_2.12-1.2.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/shapeless_2.12-2.3.7.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/shims-0.9.25.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/slf4j-api-1.7.32.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/snakeyaml-1.31.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/snappy-java-1.1.8.4.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-catalyst_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-core_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-graphx_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-kvstore_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-launcher_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-mllib_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-mllib-local_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-network-common_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-network-shuffle_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-repl_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-sketch_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-sql_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-streaming_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-tags_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-tags_2.12-3.3.2-SNAPSHOT-tests.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-unsafe_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire_2.12-0.17.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-macros_2.12-0.17.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-platform_2.12-0.17.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-util_2.12-0.17.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/stream-2.9.6.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/threeten-extra-1.5.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/tink-1.6.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/univocity-parsers-2.9.1.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/volcano-client-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/volcano-model-v1beta1-5.12.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/xbean-asm9-shaded-4.20.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/xz-1.9.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zjsonpatch-0.3.0.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zookeeper-3.6.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zookeeper-jute-3.6.2.jar
> /home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zstd-jni-1.5.2-1.jar
> /home/bjorn/test_volcano/spark/dist/jars/
> + '[' -f
> '/home/bjorn/test_volcano/spark/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar'
> ']'
> + '[' -d
> /home/bjorn/test_volcano/spark/resource-managers/kubernetes/core/target/ ']'
> + mkdir -p /home/bjorn/test_volcano/spark/dist/kubernetes/
> + cp -a
> /home/bjorn/test_volcano/spark/resource-managers/kubernetes/docker/src/main/dockerfiles
> /home/bjorn/test_volcano/spark/dist/kubernetes/
> + cp -a
> /home/bjorn/test_volcano/spark/resource-managers/kubernetes/integration-tests/tests
> /home/bjorn/test_volcano/spark/dist/kubernetes/
> + mkdir -p /home/bjorn/test_volcano/spark/dist/examples/jars
> + cp
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/aircompressor-0.21.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/annotations-17.0.0.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/commons-codec-1.15.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/commons-lang3-3.12.0.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/hive-storage-api-2.7.2.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/istack-commons-runtime-3.0.8.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/jakarta.xml.bind-api-2.3.2.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/jaxb-runtime-2.3.2.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-core-1.7.7.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-mapreduce-1.7.7.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-shims-1.7.7.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/scopt_2.12-3.7.1.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
> /home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/threeten-extra-1.5.0.jar
> /home/bjorn/test_volcano/spark/dist/examples/jars
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/aircompressor-0.21.jar
> + name=aircompressor-0.21.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/aircompressor-0.21.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/aircompressor-0.21.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/annotations-17.0.0.jar
> + name=annotations-17.0.0.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/annotations-17.0.0.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/annotations-17.0.0.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-codec-1.15.jar
> + name=commons-codec-1.15.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/commons-codec-1.15.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-codec-1.15.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-lang3-3.12.0.jar
> + name=commons-lang3-3.12.0.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/commons-lang3-3.12.0.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/commons-lang3-3.12.0.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/hive-storage-api-2.7.2.jar
> + name=hive-storage-api-2.7.2.jar
> + '[' -f
> /home/bjorn/test_volcano/spark/dist/jars/hive-storage-api-2.7.2.jar ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/hive-storage-api-2.7.2.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/istack-commons-runtime-3.0.8.jar
> + name=istack-commons-runtime-3.0.8.jar
> + '[' -f
> /home/bjorn/test_volcano/spark/dist/jars/istack-commons-runtime-3.0.8.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/istack-commons-runtime-3.0.8.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/jakarta.xml.bind-api-2.3.2.jar
> + name=jakarta.xml.bind-api-2.3.2.jar
> + '[' -f
> /home/bjorn/test_volcano/spark/dist/jars/jakarta.xml.bind-api-2.3.2.jar ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/jakarta.xml.bind-api-2.3.2.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/jaxb-runtime-2.3.2.jar
> + name=jaxb-runtime-2.3.2.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/jaxb-runtime-2.3.2.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/jaxb-runtime-2.3.2.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-core-1.7.7.jar
> + name=orc-core-1.7.7.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-core-1.7.7.jar ']'
> + rm /home/bjorn/test_volcano/spark/dist/examples/jars/orc-core-1.7.7.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-mapreduce-1.7.7.jar
> + name=orc-mapreduce-1.7.7.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-mapreduce-1.7.7.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-mapreduce-1.7.7.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/orc-shims-1.7.7.jar
> + name=orc-shims-1.7.7.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-shims-1.7.7.jar ']'
> + rm /home/bjorn/test_volcano/spark/dist/examples/jars/orc-shims-1.7.7.jar
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/scopt_2.12-3.7.1.jar
> + name=scopt_2.12-3.7.1.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/scopt_2.12-3.7.1.jar ']'
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
> + name=spark-examples_2.12-3.3.2-SNAPSHOT.jar
> + '[' -f
> /home/bjorn/test_volcano/spark/dist/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
> ']'
> + for f in "$DISTDIR"/examples/jars/*
> ++ basename
> /home/bjorn/test_volcano/spark/dist/examples/jars/threeten-extra-1.5.0.jar
> + name=threeten-extra-1.5.0.jar
> + '[' -f /home/bjorn/test_volcano/spark/dist/jars/threeten-extra-1.5.0.jar
> ']'
> + rm
> /home/bjorn/test_volcano/spark/dist/examples/jars/threeten-extra-1.5.0.jar
> + mkdir -p /home/bjorn/test_volcano/spark/dist/examples/src/main
> + cp -r /home/bjorn/test_volcano/spark/examples/src/main
> /home/bjorn/test_volcano/spark/dist/examples/src/
> + '[' -e /home/bjorn/test_volcano/spark/LICENSE-binary ']'
> + cp /home/bjorn/test_volcano/spark/LICENSE-binary
> /home/bjorn/test_volcano/spark/dist/LICENSE
> + cp -r /home/bjorn/test_volcano/spark/licenses-binary
> /home/bjorn/test_volcano/spark/dist/licenses
> + cp /home/bjorn/test_volcano/spark/NOTICE-binary
> /home/bjorn/test_volcano/spark/dist/NOTICE
> + '[' -e /home/bjorn/test_volcano/spark/CHANGES.txt ']'
> + cp -r /home/bjorn/test_volcano/spark/data
> /home/bjorn/test_volcano/spark/dist
> + '[' false == true ']'
> + echo 'Skipping building python distribution package'
> Skipping building python distribution package
> + '[' false == true ']'
> + echo 'Skipping building R source package'
> Skipping building R source package
> + mkdir /home/bjorn/test_volcano/spark/dist/conf
> + cp /home/bjorn/test_volcano/spark/conf/fairscheduler.xml.template
> /home/bjorn/test_volcano/spark/conf/log4j2.properties.template
> /home/bjorn/test_volcano/spark/conf/metrics.properties.template
> /home/bjorn/test_volcano/spark/conf/spark-defaults.conf.template
> /home/bjorn/test_volcano/spark/conf/spark-env.sh.template
> /home/bjorn/test_volcano/spark/conf/workers.template
> /home/bjorn/test_volcano/spark/dist/conf
> + cp /home/bjorn/test_volcano/spark/README.md
> /home/bjorn/test_volcano/spark/dist
> + cp -r /home/bjorn/test_volcano/spark/bin
> /home/bjorn/test_volcano/spark/dist
> + cp -r /home/bjorn/test_volcano/spark/python
> /home/bjorn/test_volcano/spark/dist
> + '[' false == true ']'
> + cp -r /home/bjorn/test_volcano/spark/sbin
> /home/bjorn/test_volcano/spark/dist
> + '[' -d /home/bjorn/test_volcano/spark/R/lib/SparkR ']'
> + '[' true == true ']'
> + TARDIR_NAME=spark-3.3.2-SNAPSHOT-bin-with-volcano
> +
> TARDIR=/home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
> + rm -rf
> /home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
> + cp -r /home/bjorn/test_volcano/spark/dist
> /home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
> + tar czf spark-3.3.2-SNAPSHOT-bin-with-volcano.tgz -C
> /home/bjorn/test_volcano/spark spark-3.3.2-SNAPSHOT-bin-with-volcano
> + rm -rf
> /home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
>
>
>
> spark]$ ls
> appveyor.yml bin build conf core dev docs
> external hadoop-cloud LICENSE licenses mllib
> NOTICE pom.xml python README.md resource-managers
> scalastyle-config.xml sql target
> assembly binder common CONTRIBUTING.md data dist examples
> graphx launcher LICENSE-binary licenses-binary mllib-local
> NOTICE-binary project R repl sbin
> spark-3.3.2-SNAPSHOT-bin-with-volcano.tgz streaming tools
>
> tor. 8. des. 2022 kl. 17:31 skrev Gnana Kumar <gn...@gmail.com>:
>
>> Hi There,
>>
>> I have been trying to follow the steps as per above url but below command
>> does not successfully create the binary distribution of spark with volcano.
>>
>> ./dev/make-distribution.sh --tgz --name with-volcano
>> -Pkubernetes,volcano,hadoop-3
>>
>> Has anyone tried it and it works ? please confirm.
>>
>> Thanks
>> Gnana
>>
>> On Tue, Nov 29, 2022 at 10:16 PM Bjørn Jørgensen <
>> bjornjorgensen@gmail.com> wrote:
>>
>>> MartinG have wrote a good howto on this
>>>
>>>
>>> https://martin-grigorov.medium.com/apache-spark-snapshot-on-kubernetes-178592613408
>>>
>>>
>>> https://martin-grigorov.medium.com/native-integration-between-apache-spark-and-volcano-kubernetes-scheduler-488f54dbbab3
>>>
>>>
>>> tir. 29. nov. 2022 kl. 16:25 skrev Gnana Kumar <gnana.kumar123@gmail.com
>>> >:
>>>
>>>> Any updates on this please ?
>>>>
>>>> On Mon, Nov 28, 2022 at 11:58 AM Gnana Kumar <gn...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes, while building the binary dist, I have used the -Pvolcano option
>>>>> but for the docker build , do we have any options to use ?
>>>>>
>>>>> >docker build -t gnana_volcano_scheduler_snapshot -f
>>>>> kubernetes/dockerfiles/spark/Dockerfile .
>>>>>
>>>>> Because my understanding is that the binary dist will have all jars
>>>>> which should be packed within the container.
>>>>>
>>>>> Please let us know how to build docker image with volcano options.
>>>>>
>>>>> Thanks
>>>>> Gnana
>>>>>
>>>>> On Sun, Nov 27, 2022 at 3:41 PM Denis Bolshakov <
>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>
>>>>>> I am talking about docker image, not about your local build. Could
>>>>>> you please make sure that docker image has a spark distro with volcano
>>>>>> support?
>>>>>>
>>>>>> вс, 27 нояб. 2022 г., 13:09 Gnana Kumar <gn...@gmail.com>:
>>>>>>
>>>>>>> Yes. I have cloned the spark source 3.3.2.SNAPSHOT and then used
>>>>>>> below command to build the binary distribution with support of volcano.
>>>>>>>
>>>>>>> >dev/./make-distribution.sh -Denforcer.skip=true -DrecompileMode=all
>>>>>>> -Pkubernetes -Pvolcano -Pscala-2.12 -DskipTests
>>>>>>>
>>>>>>> Also please find the attached screenshot of spark-kubernetes JAR
>>>>>>> built from above command and if we unzip,it has the scala class -
>>>>>>> org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> Thanks
>>>>>>> Gnana
>>>>>>>
>>>>>>> On Sun, Nov 27, 2022 at 3:23 PM Denis Bolshakov <
>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>
>>>>>>>> Does your driver docker image has a spark distro built with volcano
>>>>>>>> support?
>>>>>>>>
>>>>>>>> вс, 27 нояб. 2022 г., 11:55 Gnana Kumar <gn...@gmail.com>:
>>>>>>>>
>>>>>>>>> Actual error content is pasted below.
>>>>>>>>>
>>>>>>>>> Exception in thread "main"* java.lang.ClassNotFoundException:
>>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep*
>>>>>>>>> at
>>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>>> at
>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>>> at
>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>>>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>> at
>>>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableLike.map(TraversableLike.scala:286)
>>>>>>>>> at
>>>>>>>>> scala.collection.TraversableLike.map$(TraversableLike.scala:279)
>>>>>>>>> at
>>>>>>>>> scala.collection.AbstractTraversable.map(Traversable.scala:108)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:41)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:106)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:220)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:214)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2713)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:214)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:186)
>>>>>>>>> at org.apache.spark.deploy.SparkSubmit.org
>>>>>>>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>>>>> 22/11/27 08:35:12 INFO ShutdownHookManager: Shutdown hook called
>>>>>>>>>
>>>>>>>>> On Sun, Nov 27, 2022 at 2:11 PM Gnana Kumar <
>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Though my
>>>>>>>>>> /opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar has
>>>>>>>>>> VolcanoFeatureStep.class sparksubmit says class is not found.
>>>>>>>>>>
>>>>>>>>>> Basically I'm looking for a working example of spark submit with
>>>>>>>>>> Volcano scheduler in Kubernetes.
>>>>>>>>>>
>>>>>>>>>> Please help me resolve this issue.
>>>>>>>>>>
>>>>>>>>>> ~/spark_3.3.2/bin/spark-submit \
>>>>>>>>>> --verbose \
>>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>>>> --deploy-mode cluster \
>>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>>> --conf
>>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>>> \
>>>>>>>>>> --conf
>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>> \
>>>>>>>>>> --conf
>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>> \
>>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>>> --conf
>>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>>> \
>>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>>> --conf
>>>>>>>>>> spark.kubernetes.driver.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>>>> \
>>>>>>>>>> --conf
>>>>>>>>>> spark.kubernetes.executor.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>>>> \
>>>>>>>>>> --conf
>>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>>> --conf
>>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>>> --conf
>>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>>> --conf spark.kubernetes.file.upload.path=file:///tmp \
>>>>>>>>>> --jars
>>>>>>>>>> local:///opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar \
>>>>>>>>>>
>>>>>>>>>> local:///opt/spark/examples/jars/spark-analytics-application-1.0.jar
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> GK
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 27, 2022 at 2:03 PM Gnana Kumar <
>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, I have installed it using the command below and volcano
>>>>>>>>>>> containers are coming up successfully in my GKE cluster.
>>>>>>>>>>>
>>>>>>>>>>> >kubectl apply -f
>>>>>>>>>>> https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/installer/volcano-development.yaml
>>>>>>>>>>>
>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>
>>>>>>>>>>> However Spark submit cannot refer to the *VolcanoFeatureSetp
>>>>>>>>>>> class *and it says class has not found exception.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 27, 2022 at 2:53 AM Bjørn Jørgensen <
>>>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Have you installed volcano to the k8s cluster?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes
>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>> https://volcano.sh/en/docs/spark_on_volcano/
>>>>>>>>>>>>
>>>>>>>>>>>> lør. 26. nov. 2022 kl. 09:06 skrev Gnana Kumar <
>>>>>>>>>>>> gnana.kumar123@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> I have also attached jar which has VolcanoFeature.class.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But when I run from SparkSubmit, it says class not found.
>>>>>>>>>>>>>
>>>>>>>>>>>>> # This is required for volcano
>>>>>>>>>>>>> ~/temp/spark_3.3.2/bin/spark-submit \
>>>>>>>>>>>>> --verbose \
>>>>>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>>>>>>> --deploy-mode cluster \
>>>>>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>>>>>> \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>> \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>> \
>>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>>>>>> \
>>>>>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>>>>>> --conf
>>>>>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please help me on how to run volcano using Spark Submit.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Nov 26, 2022 at 11:43 AM Gnana Kumar <
>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> HI Denis,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is how I have built it from the parent 'spark' dir.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >mvn -Denforcer.skip=true -DrecompileMode=all -Pkubernetes
>>>>>>>>>>>>>> -Pvolcano -Pscala-2.12 -DskipTests clean package
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also once the build is completed, the generated
>>>>>>>>>>>>>> spark-kubernetes...jar has VolcanoFeature.class but when this class name is
>>>>>>>>>>>>>> referred in Spark Submit ,it gives the class not found exception.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 25, 2022 at 1:43 PM Denis Bolshakov <
>>>>>>>>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think volcano is not enabled in default build, you have to
>>>>>>>>>>>>>>> provide a maven profile by adding -Pvolcano
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, 24 Nov 2022 at 19:09, Gnana Kumar <
>>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is there any issue SNAPSHOT jars ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Nov 24, 2022 at 9:33 PM Gnana Kumar <
>>>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi There,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have cloned spark 3.3.2 snapshot source code where
>>>>>>>>>>>>>>>>> Volcano feature is present and then I built the binary distribution for the
>>>>>>>>>>>>>>>>> same.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However when I tried to use this spark 3.3.2 SNAPSHOT dist
>>>>>>>>>>>>>>>>> with Volcano but I have got class not found exception.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>>>>>>>>>> at java.base/java.lang.Class.forName0(Native
>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Any help to resolve this issue please ? Basically I would
>>>>>>>>>>>>>>>>> like to use Volcano from Spark 3.3.2 in Kubernetes cluster.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> //with Best Regards
>>>>>>>>>>>>>>> --Denis Bolshakov
>>>>>>>>>>>>>>> e-mail: bolshakov.denis@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>>>> Norge
>>>>>>>>>>>>
>>>>>>>>>>>> +47 480 94 297
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gnana
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks
>>>>>>>>>> Gnana
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Gnana
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Gnana
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Gnana
>>>>
>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>>
>> --
>> Thanks
>> Gnana
>>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
--
Thanks
Gnana
Re: Unable to run Spark Job(3.3.2 SNAPSHOT) with Volcano scheduler in Kubernetes
Posted by Bjørn Jørgensen <bj...@gmail.com>.
test_volcano]$ git clone --branch branch-3.3
https://github.com/apache/spark.git
Cloning into 'spark'...
remote: Enumerating objects: 959089, done.
remote: Counting objects: 100% (355/355), done.
remote: Compressing objects: 100% (220/220), done.
remote: Total 959089 (delta 102), reused 235 (delta 55), pack-reused 958734
Receiving objects: 100% (959089/959089), 438.07 MiB | 19.42 MiB/s, done.
Resolving deltas: 100% (426020/426020), done.
test_volcano]$ cd spark/
./dev/make-distribution.sh --tgz --name with-volcano
-Pkubernetes,volcano,hadoop-3
(...)
[INFO] Reactor Summary for Spark Project Parent POM 3.3.2-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [01:27
min]
[INFO] Spark Project Tags ................................. SUCCESS [
17.137 s]
[INFO] Spark Project Sketch ............................... SUCCESS [
4.227 s]
[INFO] Spark Project Local DB ............................. SUCCESS [
12.887 s]
[INFO] Spark Project Networking ........................... SUCCESS [
22.413 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
3.734 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [
6.212 s]
[INFO] Spark Project Launcher ............................. SUCCESS [
7.791 s]
[INFO] Spark Project Core ................................. SUCCESS [01:52
min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [
21.201 s]
[INFO] Spark Project GraphX ............................... SUCCESS [
20.063 s]
[INFO] Spark Project Streaming ............................ SUCCESS [
34.545 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:40
min]
[INFO] Spark Project SQL .................................. SUCCESS [02:47
min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:28
min]
[INFO] Spark Project Tools ................................ SUCCESS [
3.459 s]
[INFO] Spark Project Hive ................................. SUCCESS [
52.119 s]
[INFO] Spark Project REPL ................................. SUCCESS [
11.587 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [01:03
min]
[INFO] Spark Project Assembly ............................. SUCCESS [
1.484 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
11.976 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [
18.443 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [
29.297 s]
[INFO] Spark Project Examples ............................. SUCCESS [
22.312 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
6.321 s]
[INFO] Spark Avro ......................................... SUCCESS [
25.435 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 15:53 min
[INFO] Finished at: 2022-12-08T18:32:50+01:00
[INFO]
------------------------------------------------------------------------
+ rm -rf /home/bjorn/test_volcano/spark/dist
+ mkdir -p /home/bjorn/test_volcano/spark/dist/jars
+ echo 'Spark 3.3.2-SNAPSHOT (git revision 02f32ee358) built for Hadoop
3.3.2'
+ echo 'Build flags: -Pkubernetes,volcano,hadoop-3'
+ cp
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/activation-1.1.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/aircompressor-0.21.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/algebra_2.12-2.0.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/annotations-17.0.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/antlr4-runtime-4.8.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/aopalliance-repackaged-2.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arpack-2.2.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arpack_combined_all-0.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-format-7.0.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-memory-core-7.0.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-memory-netty-7.0.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/arrow-vector-7.0.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/audience-annotations-0.5.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/automaton-1.11-8.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-1.11.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-ipc-1.11.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/avro-mapred-1.11.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/blas-2.2.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/breeze_2.12-1.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/breeze-macros_2.12-1.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/cats-kernel_2.12-2.1.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/chill_2.12-0.10.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/chill-java-0.10.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-codec-1.15.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-collections-3.2.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-collections4-4.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-compiler-3.0.16.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-compress-1.21.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-crypto-1.1.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-io-2.11.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-lang-2.6.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-lang3-3.12.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-logging-1.1.3.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-math3-3.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/commons-text-1.10.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/compress-lzf-1.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/core-1.1.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-client-2.13.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-framework-2.13.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/curator-recipes-2.13.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/flatbuffers-java-1.12.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/generex-1.0.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/gson-2.8.6.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/guava-14.0.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hadoop-client-api-3.3.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hadoop-client-runtime-3.3.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hive-storage-api-2.7.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-api-2.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-locator-2.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/hk2-utils-2.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/istack-commons-runtime-3.0.8.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/ivy-2.5.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-annotations-2.13.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-core-2.13.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-databind-2.13.4.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-dataformat-yaml-2.13.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-datatype-jsr310-2.13.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jackson-module-scala_2.12-2.13.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.annotation-api-1.3.5.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.inject-2.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.servlet-api-4.0.3.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.validation-api-2.0.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.ws.rs-api-2.1.6.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jakarta.xml.bind-api-2.3.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/janino-3.0.16.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/javassist-3.25.0-GA.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jaxb-runtime-2.3.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jcl-over-slf4j-1.7.32.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-client-2.36.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-common-2.36.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-container-servlet-2.36.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-container-servlet-core-2.36.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-hk2-2.36.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jersey-server-2.36.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/JLargeArrays-1.5.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-ast_2.12-3.7.0-M11.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-core_2.12-3.7.0-M11.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-jackson_2.12-3.7.0-M11.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/json4s-scalap_2.12-3.7.0-M11.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jsr305-3.0.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/JTransforms-3.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/jul-to-slf4j-1.7.32.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kryo-shaded-4.0.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-client-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-admissionregistration-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-apiextensions-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-apps-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-autoscaling-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-batch-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-certificates-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-common-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-coordination-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-core-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-discovery-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-events-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-extensions-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-flowcontrol-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-metrics-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-networking-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-node-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-policy-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-rbac-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-scheduling-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/kubernetes-model-storageclass-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/lapack-2.2.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/leveldbjni-all-1.8.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-1.2-api-2.17.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-api-2.17.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-core-2.17.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/log4j-slf4j-impl-2.17.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/logging-interceptor-3.12.12.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/lz4-java-1.8.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-core-4.2.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-graphite-4.2.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-jmx-4.2.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-json-4.2.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/metrics-jvm-4.2.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/minlog-1.3.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-all-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-buffer-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-codec-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-common-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-handler-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-resolver-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-tcnative-classes-2.0.48.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-classes-epoll-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-classes-kqueue-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-epoll-4.1.74.Final-linux-aarch_64.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-epoll-4.1.74.Final-linux-x86_64.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-kqueue-4.1.74.Final-osx-aarch_64.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-kqueue-4.1.74.Final-osx-x86_64.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/netty-transport-native-unix-common-4.1.74.Final.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/objenesis-3.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/okhttp-3.12.12.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/okio-1.14.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/opencsv-2.3.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-core-1.7.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-mapreduce-1.7.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/orc-shims-1.7.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/oro-2.0.8.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/osgi-resource-locator-1.0.3.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/paranamer-2.8.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-column-1.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-common-1.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-encoding-1.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-format-structures-1.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-hadoop-1.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/parquet-jackson-1.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/pickle-1.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/protobuf-java-2.5.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/py4j-0.10.9.5.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/RoaringBitmap-0.9.25.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/rocksdbjni-6.20.3.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-collection-compat_2.12-2.1.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-compiler-2.12.15.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-library-2.12.15.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-parser-combinators_2.12-1.1.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-reflect-2.12.15.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/scala-xml_2.12-1.2.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/shapeless_2.12-2.3.7.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/shims-0.9.25.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/slf4j-api-1.7.32.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/snakeyaml-1.31.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/snappy-java-1.1.8.4.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-catalyst_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-core_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-graphx_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-kvstore_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-launcher_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-mllib_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-mllib-local_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-network-common_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-network-shuffle_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-repl_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-sketch_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-sql_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-streaming_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-tags_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-tags_2.12-3.3.2-SNAPSHOT-tests.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spark-unsafe_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire_2.12-0.17.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-macros_2.12-0.17.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-platform_2.12-0.17.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/spire-util_2.12-0.17.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/stream-2.9.6.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/threeten-extra-1.5.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/tink-1.6.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/univocity-parsers-2.9.1.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/volcano-client-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/volcano-model-v1beta1-5.12.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/xbean-asm9-shaded-4.20.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/xz-1.9.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zjsonpatch-0.3.0.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zookeeper-3.6.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zookeeper-jute-3.6.2.jar
/home/bjorn/test_volcano/spark/assembly/target/scala-2.12/jars/zstd-jni-1.5.2-1.jar
/home/bjorn/test_volcano/spark/dist/jars/
+ '[' -f
'/home/bjorn/test_volcano/spark/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar'
']'
+ '[' -d
/home/bjorn/test_volcano/spark/resource-managers/kubernetes/core/target/ ']'
+ mkdir -p /home/bjorn/test_volcano/spark/dist/kubernetes/
+ cp -a
/home/bjorn/test_volcano/spark/resource-managers/kubernetes/docker/src/main/dockerfiles
/home/bjorn/test_volcano/spark/dist/kubernetes/
+ cp -a
/home/bjorn/test_volcano/spark/resource-managers/kubernetes/integration-tests/tests
/home/bjorn/test_volcano/spark/dist/kubernetes/
+ mkdir -p /home/bjorn/test_volcano/spark/dist/examples/jars
+ cp
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/aircompressor-0.21.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/annotations-17.0.0.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/commons-codec-1.15.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/commons-lang3-3.12.0.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/hive-storage-api-2.7.2.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/istack-commons-runtime-3.0.8.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/jakarta.xml.bind-api-2.3.2.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/jaxb-runtime-2.3.2.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-core-1.7.7.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-mapreduce-1.7.7.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/orc-shims-1.7.7.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/scopt_2.12-3.7.1.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
/home/bjorn/test_volcano/spark/examples/target/scala-2.12/jars/threeten-extra-1.5.0.jar
/home/bjorn/test_volcano/spark/dist/examples/jars
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/aircompressor-0.21.jar
+ name=aircompressor-0.21.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/aircompressor-0.21.jar ']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/aircompressor-0.21.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/annotations-17.0.0.jar
+ name=annotations-17.0.0.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/annotations-17.0.0.jar ']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/annotations-17.0.0.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/commons-codec-1.15.jar
+ name=commons-codec-1.15.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/commons-codec-1.15.jar ']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/commons-codec-1.15.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/commons-lang3-3.12.0.jar
+ name=commons-lang3-3.12.0.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/commons-lang3-3.12.0.jar
']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/commons-lang3-3.12.0.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/hive-storage-api-2.7.2.jar
+ name=hive-storage-api-2.7.2.jar
+ '[' -f
/home/bjorn/test_volcano/spark/dist/jars/hive-storage-api-2.7.2.jar ']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/hive-storage-api-2.7.2.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/istack-commons-runtime-3.0.8.jar
+ name=istack-commons-runtime-3.0.8.jar
+ '[' -f
/home/bjorn/test_volcano/spark/dist/jars/istack-commons-runtime-3.0.8.jar
']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/istack-commons-runtime-3.0.8.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/jakarta.xml.bind-api-2.3.2.jar
+ name=jakarta.xml.bind-api-2.3.2.jar
+ '[' -f
/home/bjorn/test_volcano/spark/dist/jars/jakarta.xml.bind-api-2.3.2.jar ']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/jakarta.xml.bind-api-2.3.2.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/jaxb-runtime-2.3.2.jar
+ name=jaxb-runtime-2.3.2.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/jaxb-runtime-2.3.2.jar ']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/jaxb-runtime-2.3.2.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/orc-core-1.7.7.jar
+ name=orc-core-1.7.7.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-core-1.7.7.jar ']'
+ rm /home/bjorn/test_volcano/spark/dist/examples/jars/orc-core-1.7.7.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/orc-mapreduce-1.7.7.jar
+ name=orc-mapreduce-1.7.7.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-mapreduce-1.7.7.jar
']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/orc-mapreduce-1.7.7.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/orc-shims-1.7.7.jar
+ name=orc-shims-1.7.7.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/orc-shims-1.7.7.jar ']'
+ rm /home/bjorn/test_volcano/spark/dist/examples/jars/orc-shims-1.7.7.jar
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/scopt_2.12-3.7.1.jar
+ name=scopt_2.12-3.7.1.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/scopt_2.12-3.7.1.jar ']'
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
+ name=spark-examples_2.12-3.3.2-SNAPSHOT.jar
+ '[' -f
/home/bjorn/test_volcano/spark/dist/jars/spark-examples_2.12-3.3.2-SNAPSHOT.jar
']'
+ for f in "$DISTDIR"/examples/jars/*
++ basename
/home/bjorn/test_volcano/spark/dist/examples/jars/threeten-extra-1.5.0.jar
+ name=threeten-extra-1.5.0.jar
+ '[' -f /home/bjorn/test_volcano/spark/dist/jars/threeten-extra-1.5.0.jar
']'
+ rm
/home/bjorn/test_volcano/spark/dist/examples/jars/threeten-extra-1.5.0.jar
+ mkdir -p /home/bjorn/test_volcano/spark/dist/examples/src/main
+ cp -r /home/bjorn/test_volcano/spark/examples/src/main
/home/bjorn/test_volcano/spark/dist/examples/src/
+ '[' -e /home/bjorn/test_volcano/spark/LICENSE-binary ']'
+ cp /home/bjorn/test_volcano/spark/LICENSE-binary
/home/bjorn/test_volcano/spark/dist/LICENSE
+ cp -r /home/bjorn/test_volcano/spark/licenses-binary
/home/bjorn/test_volcano/spark/dist/licenses
+ cp /home/bjorn/test_volcano/spark/NOTICE-binary
/home/bjorn/test_volcano/spark/dist/NOTICE
+ '[' -e /home/bjorn/test_volcano/spark/CHANGES.txt ']'
+ cp -r /home/bjorn/test_volcano/spark/data
/home/bjorn/test_volcano/spark/dist
+ '[' false == true ']'
+ echo 'Skipping building python distribution package'
Skipping building python distribution package
+ '[' false == true ']'
+ echo 'Skipping building R source package'
Skipping building R source package
+ mkdir /home/bjorn/test_volcano/spark/dist/conf
+ cp /home/bjorn/test_volcano/spark/conf/fairscheduler.xml.template
/home/bjorn/test_volcano/spark/conf/log4j2.properties.template
/home/bjorn/test_volcano/spark/conf/metrics.properties.template
/home/bjorn/test_volcano/spark/conf/spark-defaults.conf.template
/home/bjorn/test_volcano/spark/conf/spark-env.sh.template
/home/bjorn/test_volcano/spark/conf/workers.template
/home/bjorn/test_volcano/spark/dist/conf
+ cp /home/bjorn/test_volcano/spark/README.md
/home/bjorn/test_volcano/spark/dist
+ cp -r /home/bjorn/test_volcano/spark/bin
/home/bjorn/test_volcano/spark/dist
+ cp -r /home/bjorn/test_volcano/spark/python
/home/bjorn/test_volcano/spark/dist
+ '[' false == true ']'
+ cp -r /home/bjorn/test_volcano/spark/sbin
/home/bjorn/test_volcano/spark/dist
+ '[' -d /home/bjorn/test_volcano/spark/R/lib/SparkR ']'
+ '[' true == true ']'
+ TARDIR_NAME=spark-3.3.2-SNAPSHOT-bin-with-volcano
+
TARDIR=/home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
+ rm -rf
/home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
+ cp -r /home/bjorn/test_volcano/spark/dist
/home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
+ tar czf spark-3.3.2-SNAPSHOT-bin-with-volcano.tgz -C
/home/bjorn/test_volcano/spark spark-3.3.2-SNAPSHOT-bin-with-volcano
+ rm -rf
/home/bjorn/test_volcano/spark/spark-3.3.2-SNAPSHOT-bin-with-volcano
spark]$ ls
appveyor.yml bin build conf core dev docs
external hadoop-cloud LICENSE licenses mllib
NOTICE pom.xml python README.md resource-managers
scalastyle-config.xml sql target
assembly binder common CONTRIBUTING.md data dist examples graphx
launcher LICENSE-binary licenses-binary mllib-local
NOTICE-binary project R repl sbin
spark-3.3.2-SNAPSHOT-bin-with-volcano.tgz streaming tools
tor. 8. des. 2022 kl. 17:31 skrev Gnana Kumar <gn...@gmail.com>:
> Hi There,
>
> I have been trying to follow the steps as per above url but below command
> does not successfully create the binary distribution of spark with volcano.
>
> ./dev/make-distribution.sh --tgz --name with-volcano
> -Pkubernetes,volcano,hadoop-3
>
> Has anyone tried it and it works ? please confirm.
>
> Thanks
> Gnana
>
> On Tue, Nov 29, 2022 at 10:16 PM Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> MartinG have wrote a good howto on this
>>
>>
>> https://martin-grigorov.medium.com/apache-spark-snapshot-on-kubernetes-178592613408
>>
>>
>> https://martin-grigorov.medium.com/native-integration-between-apache-spark-and-volcano-kubernetes-scheduler-488f54dbbab3
>>
>>
>> tir. 29. nov. 2022 kl. 16:25 skrev Gnana Kumar <gnana.kumar123@gmail.com
>> >:
>>
>>> Any updates on this please ?
>>>
>>> On Mon, Nov 28, 2022 at 11:58 AM Gnana Kumar <gn...@gmail.com>
>>> wrote:
>>>
>>>> Yes, while building the binary dist, I have used the -Pvolcano option
>>>> but for the docker build , do we have any options to use ?
>>>>
>>>> >docker build -t gnana_volcano_scheduler_snapshot -f
>>>> kubernetes/dockerfiles/spark/Dockerfile .
>>>>
>>>> Because my understanding is that the binary dist will have all jars
>>>> which should be packed within the container.
>>>>
>>>> Please let us know how to build docker image with volcano options.
>>>>
>>>> Thanks
>>>> Gnana
>>>>
>>>> On Sun, Nov 27, 2022 at 3:41 PM Denis Bolshakov <
>>>> bolshakov.denis@gmail.com> wrote:
>>>>
>>>>> I am talking about docker image, not about your local build. Could you
>>>>> please make sure that docker image has a spark distro with volcano support?
>>>>>
>>>>> вс, 27 нояб. 2022 г., 13:09 Gnana Kumar <gn...@gmail.com>:
>>>>>
>>>>>> Yes. I have cloned the spark source 3.3.2.SNAPSHOT and then used
>>>>>> below command to build the binary distribution with support of volcano.
>>>>>>
>>>>>> >dev/./make-distribution.sh -Denforcer.skip=true -DrecompileMode=all
>>>>>> -Pkubernetes -Pvolcano -Pscala-2.12 -DskipTests
>>>>>>
>>>>>> Also please find the attached screenshot of spark-kubernetes JAR
>>>>>> built from above command and if we unzip,it has the scala class -
>>>>>> org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Thanks
>>>>>> Gnana
>>>>>>
>>>>>> On Sun, Nov 27, 2022 at 3:23 PM Denis Bolshakov <
>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>
>>>>>>> Does your driver docker image has a spark distro built with volcano
>>>>>>> support?
>>>>>>>
>>>>>>> вс, 27 нояб. 2022 г., 11:55 Gnana Kumar <gn...@gmail.com>:
>>>>>>>
>>>>>>>> Actual error content is pasted below.
>>>>>>>>
>>>>>>>> Exception in thread "main"* java.lang.ClassNotFoundException:
>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep*
>>>>>>>> at
>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>> at
>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>> at
>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>> at
>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>> at
>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>> at
>>>>>>>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>>>>>>>> at
>>>>>>>> scala.collection.TraversableLike.map(TraversableLike.scala:286)
>>>>>>>> at
>>>>>>>> scala.collection.TraversableLike.map$(TraversableLike.scala:279)
>>>>>>>> at
>>>>>>>> scala.collection.AbstractTraversable.map(Traversable.scala:108)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:41)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:106)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:220)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:214)
>>>>>>>> at
>>>>>>>> org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2713)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:214)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:186)
>>>>>>>> at org.apache.spark.deploy.SparkSubmit.org
>>>>>>>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>>>>>>>> at
>>>>>>>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>>>> 22/11/27 08:35:12 INFO ShutdownHookManager: Shutdown hook called
>>>>>>>>
>>>>>>>> On Sun, Nov 27, 2022 at 2:11 PM Gnana Kumar <
>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Though my
>>>>>>>>> /opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar has
>>>>>>>>> VolcanoFeatureStep.class sparksubmit says class is not found.
>>>>>>>>>
>>>>>>>>> Basically I'm looking for a working example of spark submit with
>>>>>>>>> Volcano scheduler in Kubernetes.
>>>>>>>>>
>>>>>>>>> Please help me resolve this issue.
>>>>>>>>>
>>>>>>>>> ~/spark_3.3.2/bin/spark-submit \
>>>>>>>>> --verbose \
>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>>> --deploy-mode cluster \
>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>> --conf
>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>> \
>>>>>>>>> --conf
>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>> \
>>>>>>>>> --conf
>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>> \
>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>> --conf
>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>> \
>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>> --conf
>>>>>>>>> spark.kubernetes.driver.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>>> \
>>>>>>>>> --conf
>>>>>>>>> spark.kubernetes.executor.container.image=gnana_volcano_scheduler_snapshot:latest
>>>>>>>>> \
>>>>>>>>> --conf
>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>> --conf
>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>> --conf
>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>> --conf spark.kubernetes.file.upload.path=file:///tmp \
>>>>>>>>> --jars
>>>>>>>>> local:///opt/spark/jars/spark-kubernetes_2.12-3.3.2-SNAPSHOT.jar \
>>>>>>>>>
>>>>>>>>> local:///opt/spark/examples/jars/spark-analytics-application-1.0.jar
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> GK
>>>>>>>>>
>>>>>>>>> On Sun, Nov 27, 2022 at 2:03 PM Gnana Kumar <
>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, I have installed it using the command below and volcano
>>>>>>>>>> containers are coming up successfully in my GKE cluster.
>>>>>>>>>>
>>>>>>>>>> >kubectl apply -f
>>>>>>>>>> https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/installer/volcano-development.yaml
>>>>>>>>>>
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> However Spark submit cannot refer to the *VolcanoFeatureSetp
>>>>>>>>>> class *and it says class has not found exception.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Gnana
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 27, 2022 at 2:53 AM Bjørn Jørgensen <
>>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Have you installed volcano to the k8s cluster?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes
>>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>>>>> https://volcano.sh/en/docs/spark_on_volcano/
>>>>>>>>>>>
>>>>>>>>>>> lør. 26. nov. 2022 kl. 09:06 skrev Gnana Kumar <
>>>>>>>>>>> gnana.kumar123@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> I have also attached jar which has VolcanoFeature.class.
>>>>>>>>>>>>
>>>>>>>>>>>> But when I run from SparkSubmit, it says class not found.
>>>>>>>>>>>>
>>>>>>>>>>>> # This is required for volcano
>>>>>>>>>>>> ~/temp/spark_3.3.2/bin/spark-submit \
>>>>>>>>>>>> --verbose \
>>>>>>>>>>>> --class com.demo.spark.SpringBootStarter \
>>>>>>>>>>>> --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>>>>>>>>>>>> --deploy-mode cluster \
>>>>>>>>>>>> --name sparkSampleApp2 \
>>>>>>>>>>>> --conf spark.kubernetes.namespace=default \
>>>>>>>>>>>> --conf spark.network.timeout=300 \
>>>>>>>>>>>> --conf spark.executor.instances=1 \
>>>>>>>>>>>> --conf spark.kubernetes.scheduler.name=volcano \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/home/gnana_kumar123/spark/volcano_spark_podgroup_template_low_priority.yaml
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.size=3 \
>>>>>>>>>>>> --conf spark.kubernetes.allocation.batch.delay=1 \
>>>>>>>>>>>> --conf spark.driver.cores=1 \
>>>>>>>>>>>> --conf spark.executor.cores=1 \
>>>>>>>>>>>> --conf spark.driver.memory=500m \
>>>>>>>>>>>> --conf spark.executor.memory=500m \
>>>>>>>>>>>> --conf spark.dynamicAllocation.enabled=true \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.dynamicAllocation.shuffleTracking.enabled=true \
>>>>>>>>>>>> --conf spark.kubernetes.driver.pod.name=spark-election-driver
>>>>>>>>>>>> \
>>>>>>>>>>>> --conf spark.app.name=spark-election \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
>>>>>>>>>>>> --conf
>>>>>>>>>>>> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"\
>>>>>>>>>>>>
>>>>>>>>>>>> Please help me on how to run volcano using Spark Submit.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gnana
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Nov 26, 2022 at 11:43 AM Gnana Kumar <
>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> HI Denis,
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is how I have built it from the parent 'spark' dir.
>>>>>>>>>>>>>
>>>>>>>>>>>>> >mvn -Denforcer.skip=true -DrecompileMode=all -Pkubernetes
>>>>>>>>>>>>> -Pvolcano -Pscala-2.12 -DskipTests clean package
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also once the build is completed, the generated
>>>>>>>>>>>>> spark-kubernetes...jar has VolcanoFeature.class but when this class name is
>>>>>>>>>>>>> referred in Spark Submit ,it gives the class not found exception.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Nov 25, 2022 at 1:43 PM Denis Bolshakov <
>>>>>>>>>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think volcano is not enabled in default build, you have to
>>>>>>>>>>>>>> provide a maven profile by adding -Pvolcano
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, 24 Nov 2022 at 19:09, Gnana Kumar <
>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is there any issue SNAPSHOT jars ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Nov 24, 2022 at 9:33 PM Gnana Kumar <
>>>>>>>>>>>>>>> gnana.kumar123@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi There,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have cloned spark 3.3.2 snapshot source code where
>>>>>>>>>>>>>>>> Volcano feature is present and then I built the binary distribution for the
>>>>>>>>>>>>>>>> same.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However when I tried to use this spark 3.3.2 SNAPSHOT dist
>>>>>>>>>>>>>>>> with Volcano but I have got class not found exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>>> spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>> --conf
>>>>>>>>>>>>>>>> spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>>>>>>> java.lang.ClassNotFoundException:
>>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.features.VolcanoFeatureStep
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>>>>>>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.apache.spark.util.Utils$.classForName(Utils.scala:216)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:42)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any help to resolve this issue please ? Basically I would
>>>>>>>>>>>>>>>> like to use Volcano from Spark 3.3.2 in Kubernetes cluster.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> //with Best Regards
>>>>>>>>>>>>>> --Denis Bolshakov
>>>>>>>>>>>>>> e-mail: bolshakov.denis@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gnana
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gnana
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>>> Norge
>>>>>>>>>>>
>>>>>>>>>>> +47 480 94 297
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks
>>>>>>>>>> Gnana
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks
>>>>>>>>> Gnana
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks
>>>>>>>> Gnana
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks
>>>>>> Gnana
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Gnana
>>>>
>>>
>>>
>>> --
>>> Thanks
>>> Gnana
>>>
>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
>
> --
> Thanks
> Gnana
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297