You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Iain Morrison (Jira)" <ji...@apache.org> on 2023/07/03 13:53:00 UTC
[jira] [Commented] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location
[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739622#comment-17739622 ]
Iain Morrison commented on SPARK-44152:
---------------------------------------
I hit this same issue. I believe the the change in behaviour was introduced in:
{quote}SPARK-43540 - Add working directory into classpath on the driver in K8S cluster mode
{quote}
Our docker file was overriding the working directory of the base spark image
{code:java}
FROM apache/spark:3.4.1@sha256:a1dd2487a97fb5e35c5a5b409e830b501a92919029c62f9a559b13c4f5c50f63 as image
WORKDIR /spark-jars
COPY --from=build /...../target/scala-2.12/my-spark.jar /spark-jars/ {code}
Changing it to this solved the problem:
{code:java}
FROM apache/spark:3.4.1@sha256:a1dd2487a97fb5e35c5a5b409e830b501a92919029c62f9a559b13c4f5c50f63 as image
USER root
RUN mkdir /spark-jars
USER spark
COPY --from=build /...../target/scala-2.12/my-spark.jar /spark-jars/ {code}
So probably not an issue.. Is there any documentation/guidance to follow??
> Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-44152
> URL: https://issues.apache.org/jira/browse/SPARK-44152
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
>
> I have a spark application that is deployed using k8s and it is of version 3.3.2 Recently there were some vulneabilities in spark 3.3.2
> I changed my dockerfile to download 3.4.0 instead of 3.3.2 and also my application jar is built on spark 3.4.0
> However while deploying, I get this error
>
> *{{Exception in thread "main" java.nio.file.NoSuchFileException: <path>/spark-assembly-1.0.jar}}*
>
> I have this in deployment.yaml of the app
>
> *mainApplicationFile: "local:///<path>/spark-assembly-1.0.jar"*
>
>
>
>
> and I have not changed anything related to that. I see that some code has changed in spark 3.4.0 core's source code regarding jar location.
> Has it really changed the functionality ? Is there anyone who is facing same issue as me ? Should the path be specified in a different way?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org