You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jatin Puri <ja...@microsoft.com.INVALID> on 2021/08/27 12:57:31 UTC
Classpath conflicts.
Consider this scenario.
We submit a spark job using `spark-submit` with a list of dependencies. Consider the dependency as `HelloWorld-1.0.jar`.
Later, another job is submitted with an updated jar `HelloWorld-2.0.jar`. Will both the jobs have their own respective different classpaths? Or will the version of HelloWorld jars be available for both jobs and future jobs?
========================================================================================
Above is wrt Spark. Now when it comes to databricks submit-job from UI, I find the behavior a bit worrying. When I submit the jobs with Azure Databricks, I find the older dependency jars (of previous jobs), still available on the classpath of the newly submitted jobs. Leading to inconsistencies.
[cid:4763cfa3-d8ca-4943-909b-47a554afecfe]
What is the recommended way, where the dependency jars of the jar-to-be-submitted changes drastically? Spawn a new cluster each time there is a change? Is submitting a job in databricks runtime different from `spark-submit`?
Regards,
Jatin
Re: [EXTERNAL] Re: Classpath conflicts.
Posted by Jatin Puri <ja...@microsoft.com.INVALID>.
Thanks Sean. That helps with clearing regarding spark submit.
With Azure databricks, I was resubmitting job with jars of different versions. But it seems to keep piling up jars across all jobs and adding to the classpath of every new job.
Thanks again.
________________________________
From: Sean Owen <sr...@gmail.com>
Sent: Friday, August 27, 2021 7:08:19 PM
To: Jatin Puri <ja...@microsoft.com>
Cc: user@spark.apache.org <us...@spark.apache.org>
Subject: [EXTERNAL] Re: Classpath conflicts.
spark-submit creates a new application with its own classpath.
Your question is specific to Databricks, so I'd follow up there. It seems like you are attaching different copies of the same library to an existing cluster, which isn't what you want.
On Fri, Aug 27, 2021 at 7:58 AM Jatin Puri <ja...@microsoft.com.invalid> wrote:
Consider this scenario.
We submit a spark job using `spark-submit` with a list of dependencies. Consider the dependency as `HelloWorld-1.0.jar`.
Later, another job is submitted with an updated jar `HelloWorld-2.0.jar`. Will both the jobs have their own respective different classpaths? Or will the version of HelloWorld jars be available for both jobs and future jobs?
========================================================================================
Above is wrt Spark. Now when it comes to databricks submit-job from UI, I find the behavior a bit worrying. When I submit the jobs with Azure Databricks, I find the older dependency jars (of previous jobs), still available on the classpath of the newly submitted jobs. Leading to inconsistencies.
[cid:17b87d44b4bcb971f161]
What is the recommended way, where the dependency jars of the jar-to-be-submitted changes drastically? Spawn a new cluster each time there is a change? Is submitting a job in databricks runtime different from `spark-submit`?
Regards,
Jatin
Re: Classpath conflicts.
Posted by Sean Owen <sr...@gmail.com>.
spark-submit creates a new application with its own classpath.
Your question is specific to Databricks, so I'd follow up there. It seems
like you are attaching different copies of the same library to an existing
cluster, which isn't what you want.
On Fri, Aug 27, 2021 at 7:58 AM Jatin Puri <ja...@microsoft.com.invalid>
wrote:
>
> Consider this scenario.
>
> We submit a spark job using `spark-submit` with a list of dependencies.
> Consider the dependency as `HelloWorld-1.0.jar`.
>
> Later, another job is submitted with an updated jar `HelloWorld-2.0.jar`.
> Will both the jobs have their own respective different classpaths? Or will
> the version of HelloWorld jars be available for both jobs and future jobs?
>
>
> ========================================================================================
>
> Above is wrt Spark. Now when it comes to databricks submit-job from UI, I
> find the behavior a bit worrying. When I submit the jobs with Azure
> Databricks, I find the older dependency jars (of previous jobs), still
> available on the classpath of the newly submitted jobs. Leading to
> inconsistencies.
>
>
> What is the recommended way, where the dependency jars of the
> jar-to-be-submitted changes drastically? Spawn a new cluster each time
> there is a change? Is submitting a job in databricks runtime different from
> `spark-submit`?
>
> Regards,
> Jatin
>