You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Federico D'Ambrosio <fe...@gmail.com> on 2019/03/18 11:34:14 UTC
Spark does not load all classes in fat jar
Hello everyone,
We're having a serious issue, where we get ClassNotFoundException because,
apparently the class is not found within the classpath of Spark, in both
the Driver and Executors.
First, I checked whether the class was actually within the jar with jar tf,
and there actually is. Then, I activated the following options to see which
classes are actually loaded:
--conf 'spark.driver.extraJavaOptions=-verbose:class' --conf
'spark.executor.extraJavaOptions=-verbose:class'
and I can see from the YARN stdout logs that some classes, just like the
one throwing the exception, are not actually being loaded while other
classes are.
I tried, then, using --jars to pass the jar containing the missing classes,
and also using addJar() from the spark context, to no avail.
This looks like an issue with Spark class loader.
Any idea about what's happenig here? I'm using Spark 2.3.1.3.0.0.0-1634
(HDP 3.0).
Thank you for your help,
Federico
Re: Spark does not load all classes in fat jar
Posted by Federico D'Ambrosio <fe...@gmail.com>.
Hi Jorn, thank you for your response.
I'm sorry I didn't mention it in the previous mail, the class name is
'sherlogic_mon_interface', a class of our project, which is intended to be
instantiated dinamically (Class.forName(" sherlogic_mon_interface")) within
an executor. So, it's not within the Spark core library.
To give a bit more context, we've got 3 modules, let's call them: core, ext
and util. The missing classes are *some* (and that's really weird, not all
of them) of the classes defined in the two modules ext and util.
The weird thing is that we're specifically building the uberjar with all
those dependencies, just like you said, because we thought it was related
to an incorrect resolution of the additional jars, so we added those
modules as dependencies explicitly (checked again with jar tf, classes are
present in the jar). And still, the same issue of ClassNotFoundException.
Feels like I'm missing something.
Thanks,
Federico
Il giorno lun 18 mar 2019 alle ore 14:09 Jörn Franke <jo...@gmail.com>
ha scritto:
> Fat jar with shading as the application not as an additional jar package
>
> Am 18.03.2019 um 14:08 schrieb Jörn Franke <jo...@gmail.com>:
>
> Maybe that class is already loaded as part of a core library of Spark?
>
> Do you have concrete class names?
>
> In doubt create a fat jar and shade the dependencies in question
>
> Am 18.03.2019 um 12:34 schrieb Federico D'Ambrosio <fe...@gmail.com>:
>
> Hello everyone,
>
> We're having a serious issue, where we get ClassNotFoundException because,
> apparently the class is not found within the classpath of Spark, in both
> the Driver and Executors.
>
> First, I checked whether the class was actually within the jar with jar tf,
> and there actually is. Then, I activated the following options to see
> which classes are actually loaded:
>
> --conf 'spark.driver.extraJavaOptions=-verbose:class' --conf
> 'spark.executor.extraJavaOptions=-verbose:class'
>
> and I can see from the YARN stdout logs that some classes, just like the
> one throwing the exception, are not actually being loaded while other
> classes are.
> I tried, then, using --jars to pass the jar containing the missing
> classes, and also using addJar() from the spark context, to no avail.
>
> This looks like an issue with Spark class loader.
>
> Any idea about what's happenig here? I'm using Spark 2.3.1.3.0.0.0-1634
> (HDP 3.0).
>
> Thank you for your help,
> Federico
>
>
--
Federico D'Ambrosio
Re: Spark does not load all classes in fat jar
Posted by Jörn Franke <jo...@gmail.com>.
Fat jar with shading as the application not as an additional jar package
> Am 18.03.2019 um 14:08 schrieb Jörn Franke <jo...@gmail.com>:
>
> Maybe that class is already loaded as part of a core library of Spark?
>
> Do you have concrete class names?
>
> In doubt create a fat jar and shade the dependencies in question
>
>> Am 18.03.2019 um 12:34 schrieb Federico D'Ambrosio <fe...@gmail.com>:
>>
>> Hello everyone,
>>
>> We're having a serious issue, where we get ClassNotFoundException because, apparently the class is not found within the classpath of Spark, in both the Driver and Executors.
>>
>> First, I checked whether the class was actually within the jar with jar tf, and there actually is. Then, I activated the following options to see which classes are actually loaded:
>>
>> --conf 'spark.driver.extraJavaOptions=-verbose:class' --conf 'spark.executor.extraJavaOptions=-verbose:class'
>>
>> and I can see from the YARN stdout logs that some classes, just like the one throwing the exception, are not actually being loaded while other classes are.
>> I tried, then, using --jars to pass the jar containing the missing classes, and also using addJar() from the spark context, to no avail.
>>
>> This looks like an issue with Spark class loader.
>>
>> Any idea about what's happenig here? I'm using Spark 2.3.1.3.0.0.0-1634 (HDP 3.0).
>>
>> Thank you for your help,
>> Federico
Re: Spark does not load all classes in fat jar
Posted by Jörn Franke <jo...@gmail.com>.
Maybe that class is already loaded as part of a core library of Spark?
Do you have concrete class names?
In doubt create a fat jar and shade the dependencies in question
> Am 18.03.2019 um 12:34 schrieb Federico D'Ambrosio <fe...@gmail.com>:
>
> Hello everyone,
>
> We're having a serious issue, where we get ClassNotFoundException because, apparently the class is not found within the classpath of Spark, in both the Driver and Executors.
>
> First, I checked whether the class was actually within the jar with jar tf, and there actually is. Then, I activated the following options to see which classes are actually loaded:
>
> --conf 'spark.driver.extraJavaOptions=-verbose:class' --conf 'spark.executor.extraJavaOptions=-verbose:class'
>
> and I can see from the YARN stdout logs that some classes, just like the one throwing the exception, are not actually being loaded while other classes are.
> I tried, then, using --jars to pass the jar containing the missing classes, and also using addJar() from the spark context, to no avail.
>
> This looks like an issue with Spark class loader.
>
> Any idea about what's happenig here? I'm using Spark 2.3.1.3.0.0.0-1634 (HDP 3.0).
>
> Thank you for your help,
> Federico