You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jan Botorek <Ja...@infor.com> on 2016/11/01 12:11:45 UTC
Add jar files on classpath when submitting tasks to Spark
Hello,
I have a problem trying to add jar files to be available on classpath when submitting task to Spark.
In my spark-defaults.conf file I have configuration:
spark.driver.extraClassPath = path/to/folder/with/jars
all jars in the folder are available in SPARK-SHELL
The problem is that jars are not on the classpath for SPARK-MASTER; more precisely - when I submit any job that utilizes any jar from external folder, the java.lang.ClassNotFoundException is thrown.
Moving all external jars into the jars folder solves the situation, but we need to keep external files separatedly.
Thank you for any help
Best regards,
Jan
Re: Add jar files on classpath when submitting tasks to Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Well if you look at the spark-shell script this is what it says
# SPARK-4161: scala does not assume use of the java classpath,
# so we need to add the "-Dscala.usejavacp=true" flag manually. We
# do this specifically for the Spark shell because the scala REPL
# has its own class loader, and any additional classpath specified
# *through spark.driver.extraClassPath is not automatically propagated.*
Whether this is relevant or not I am not sure
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 2 November 2016 at 09:36, Jan Botorek <Ja...@infor.com> wrote:
> Thank you for the example.
>
> I am able to submit the task when using the –jars parameter as followed:
>
>
>
> *spark-submit --class com.infor.skyvault.tests.LinearRegressionTest
> --master local –jars path/to/jar/one;path/to/jar/two
> C:\_resources\spark-1.0-SNAPSHOT.jar -DtrainDataPath="/path/to/model/data"*
>
>
>
> But, I would like to find out, why the setting of
> *spark.driver.extraClassPath* attribute in spark-defaults.xml is not
> applied when submitting the task.
>
> In our scenario let’s assume that all workers (currently only one worker)
> have the attribute *spark.driver.extraClassPath* set to the same path and
> the folder on all workers contains the same .jar files.
>
>
>
> Thank you for your help,
>
>
>
> Regards,
>
> Jan
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Sent:* Tuesday, November 1, 2016 3:22 PM
>
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* Vinod Mangipudi <vi...@gmail.com>; user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> If you are using local mode then there is only one JVM. In Linux as below
> mine looks like this
>
>
>
> ${SPARK_HOME}/bin/spark-submit \
> --packages ${PACKAGES} \
> --driver-memory 8G \
> --num-executors 1 \
> --executor-memory 8G \
> *--master local[12] \*
> --conf "${SCHEDULER}" \
> --conf "${EXTRAJAVAOPTIONS}" \
> --jars ${JARS} \
> --class "${FILE_NAME}" \
> --conf "${SPARKUIPORT}" \
> --conf "${SPARKDRIVERPORT}" \
> --conf "${SPARKFILESERVERPORT}" \
> --conf "${SPARKBLOCKMANAGERPORT}" \
> --conf "${SPARKKRYOSERIALIZERBUFFERMAX}" \
> ${JAR_FILE}
>
>
>
> These parameters are defined below
>
> function default_settings {
> export PACKAGES="com.databricks:spark-csv_2.11:1.3.0"
> export SCHEDULER="spark.scheduler.mode=FAIR"
> export EXTRAJAVAOPTIONS="spark.executor.extraJavaOptions=-XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps"
> export JARS="/home/hduser/jars/spark-streaming-kafka-assembly_2.11-
> 1.6.1.jar"
> export SPARKUIPORT="spark.ui.port=55555"
> export SPARKDRIVERPORT="spark.driver.port=54631"
> export SPARKFILESERVERPORT="spark.fileserver.port=54731"
> export SPARKBLOCKMANAGERPORT="spark.blockManager.port=54832"
> export SPARKKRYOSERIALIZERBUFFERMAX="spark.kryoserializer.buffer.max=512"
> }
>
>
>
> and other jar files have passed through --jars. Note that ${JAR_FILE} in
> my case is built through MVN or SBT
>
>
>
> HTH
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
> On 1 November 2016 at 14:02, Jan Botorek <Ja...@infor.com> wrote:
>
> Yes, exactly.
> My (testing) run script is:
>
> spark-submit --class com.infor.skyvault.tests.LinearRegressionTest
> --master local C:\_resources\spark-1.0-SNAPSHOT.jar
> -DtrainDataPath="/path/to/model/data"
>
>
>
>
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Sent:* Tuesday, November 1, 2016 2:51 PM
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* Vinod Mangipudi <vi...@gmail.com>; user <us...@spark.apache.org>
>
>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> Are you submitting your job through spark-submit?
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
> On 1 November 2016 at 13:39, Jan Botorek <Ja...@infor.com> wrote:
>
> Hello,
>
> This approach unfortunately doesn’t work for job submission for me. It
> works in the shell, but not when submitted.
>
> I ensured the (only worker) node has desired directory.
>
>
>
> Neither specifying all jars as you suggested, neither using
> */path/to/jarfiles/** works.
>
>
>
> Could you verify, that using this settings you are able to submit jobs
> with according dependencies, please?
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Sent:* Tuesday, November 1, 2016 2:18 PM
> *To:* Vinod Mangipudi <vi...@gmail.com>
>
>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> you can do that as long as every node has the directory referenced.
>
>
>
> For example
>
>
>
> spark.driver.extraClassPath /home/hduser/jars/ojdbc6.jar:/
> home/hduser/jars/jconn4.jar
> spark.executor.extraClassPath /home/hduser/jars/ojdbc6.jar:/
> home/hduser/jars/jconn4.jar
>
>
>
> this will work as long as all nodes have that directory.
>
>
>
> The other alternative is to mount the shared directory as NFS mount across
> all the nodes and all the noses can read from that shared directory
>
>
>
> HTH
>
>
>
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
> On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com> wrote:
>
> unsubscribe
>
>
>
> On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com> wrote:
>
> Thank you for the reply.
>
> I am aware of the parameters used when submitting the tasks (--jars is
> working for us).
>
>
>
> But, isn’t there any way how to specify a location (directory) for jars
> „in global“ - in the spark-defaults.conf??
>
>
>
>
>
> *From:* ayan guha [mailto:guha.ayan@gmail.com]
> *Sent:* Tuesday, November 1, 2016 1:49 PM
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> There are options to specify external jars in the form of --jars,
> --driver-classpath etc depending on spark version and cluster manager..
> Please see spark documents for configuration sections and/or run spark
> submit help to see available options.
>
> On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com> wrote:
>
> Hello,
>
> I have a problem trying to add jar files to be available on classpath when
> submitting task to Spark.
>
>
>
> In my spark-defaults.conf file I have configuration:
>
> *spark.driver.extraClassPath = path/to/folder/with/jars*
>
> all jars in the folder are available in SPARK-SHELL
>
>
>
> The problem is that jars are not on the classpath for SPARK-MASTER; more
> precisely – when I submit any job that utilizes any jar from external
> folder, the* java.lang.ClassNotFoundException* is thrown.
>
> Moving all external jars into the *jars* folder solves the situation, but
> we need to keep external files separatedly.
>
>
>
> Thank you for any help
>
> Best regards,
>
> Jan
>
>
>
>
>
>
>
>
>
RE: Add jar files on classpath when submitting tasks to Spark
Posted by Jan Botorek <Ja...@infor.com>.
Thank you for the example.
I am able to submit the task when using the –jars parameter as followed:
spark-submit --class com.infor.skyvault.tests.LinearRegressionTest --master local –jars path/to/jar/one;path/to/jar/two C:\_resources\spark-1.0-SNAPSHOT.jar -DtrainDataPath="/path/to/model/data"
But, I would like to find out, why the setting of spark.driver.extraClassPath attribute in spark-defaults.xml is not applied when submitting the task.
In our scenario let’s assume that all workers (currently only one worker) have the attribute spark.driver.extraClassPath set to the same path and the folder on all workers contains the same .jar files.
Thank you for your help,
Regards,
Jan
From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
Sent: Tuesday, November 1, 2016 3:22 PM
To: Jan Botorek <Ja...@infor.com>
Cc: Vinod Mangipudi <vi...@gmail.com>; user <us...@spark.apache.org>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
If you are using local mode then there is only one JVM. In Linux as below mine looks like this
${SPARK_HOME}/bin/spark-submit \
--packages ${PACKAGES} \
--driver-memory 8G \
--num-executors 1 \
--executor-memory 8G \
--master local[12] \
--conf "${SCHEDULER}" \
--conf "${EXTRAJAVAOPTIONS}" \
--jars ${JARS} \
--class "${FILE_NAME}" \
--conf "${SPARKUIPORT}" \
--conf "${SPARKDRIVERPORT}" \
--conf "${SPARKFILESERVERPORT}" \
--conf "${SPARKBLOCKMANAGERPORT}" \
--conf "${SPARKKRYOSERIALIZERBUFFERMAX}" \
${JAR_FILE}
These parameters are defined below
function default_settings {
export PACKAGES="com.databricks:spark-csv_2.11:1.3.0"
export SCHEDULER="spark.scheduler.mode=FAIR"
export EXTRAJAVAOPTIONS="spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
export JARS="/home/hduser/jars/spark-streaming-kafka-assembly_2.11-1.6.1.jar"
export SPARKUIPORT="spark.ui.port=55555"
export SPARKDRIVERPORT="spark.driver.port=54631"
export SPARKFILESERVERPORT="spark.fileserver.port=54731"
export SPARKBLOCKMANAGERPORT="spark.blockManager.port=54832"
export SPARKKRYOSERIALIZERBUFFERMAX="spark.kryoserializer.buffer.max=512"
}
and other jar files have passed through --jars. Note that ${JAR_FILE} in my case is built through MVN or SBT
HTH
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 1 November 2016 at 14:02, Jan Botorek <Ja...@infor.com>> wrote:
Yes, exactly.
My (testing) run script is:
spark-submit --class com.infor.skyvault.tests.LinearRegressionTest --master local C:\_resources\spark-1.0-SNAPSHOT.jar -DtrainDataPath="/path/to/model/data"
From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com<ma...@gmail.com>]
Sent: Tuesday, November 1, 2016 2:51 PM
To: Jan Botorek <Ja...@infor.com>>
Cc: Vinod Mangipudi <vi...@gmail.com>>; user <us...@spark.apache.org>>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
Are you submitting your job through spark-submit?
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 1 November 2016 at 13:39, Jan Botorek <Ja...@infor.com>> wrote:
Hello,
This approach unfortunately doesn’t work for job submission for me. It works in the shell, but not when submitted.
I ensured the (only worker) node has desired directory.
Neither specifying all jars as you suggested, neither using /path/to/jarfiles/* works.
Could you verify, that using this settings you are able to submit jobs with according dependencies, please?
From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com<ma...@gmail.com>]
Sent: Tuesday, November 1, 2016 2:18 PM
To: Vinod Mangipudi <vi...@gmail.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
you can do that as long as every node has the directory referenced.
For example
spark.driver.extraClassPath /home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
spark.executor.extraClassPath /home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
this will work as long as all nodes have that directory.
The other alternative is to mount the shared directory as NFS mount across all the nodes and all the noses can read from that shared directory
HTH
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com>> wrote:
unsubscribe
On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com>> wrote:
Thank you for the reply.
I am aware of the parameters used when submitting the tasks (--jars is working for us).
But, isn’t there any way how to specify a location (directory) for jars „in global“ - in the spark-defaults.conf??
From: ayan guha [mailto:guha.ayan@gmail.com<ma...@gmail.com>]
Sent: Tuesday, November 1, 2016 1:49 PM
To: Jan Botorek <Ja...@infor.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
There are options to specify external jars in the form of --jars, --driver-classpath etc depending on spark version and cluster manager.. Please see spark documents for configuration sections and/or run spark submit help to see available options.
On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com>> wrote:
Hello,
I have a problem trying to add jar files to be available on classpath when submitting task to Spark.
In my spark-defaults.conf file I have configuration:
spark.driver.extraClassPath = path/to/folder/with/jars
all jars in the folder are available in SPARK-SHELL
The problem is that jars are not on the classpath for SPARK-MASTER; more precisely – when I submit any job that utilizes any jar from external folder, the java.lang.ClassNotFoundException is thrown.
Moving all external jars into the jars folder solves the situation, but we need to keep external files separatedly.
Thank you for any help
Best regards,
Jan
Re: Add jar files on classpath when submitting tasks to Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
If you are using local mode then there is only one JVM. In Linux as below
mine looks like this
${SPARK_HOME}/bin/spark-submit \
--packages ${PACKAGES} \
--driver-memory 8G \
--num-executors 1 \
--executor-memory 8G \
*--master local[12] \* --conf "${SCHEDULER}" \
--conf "${EXTRAJAVAOPTIONS}" \
--jars ${JARS} \
--class "${FILE_NAME}" \
--conf "${SPARKUIPORT}" \
--conf "${SPARKDRIVERPORT}" \
--conf "${SPARKFILESERVERPORT}" \
--conf "${SPARKBLOCKMANAGERPORT}" \
--conf "${SPARKKRYOSERIALIZERBUFFERMAX}" \
${JAR_FILE}
These parameters are defined below
function default_settings {
export PACKAGES="com.databricks:spark-csv_2.11:1.3.0"
export SCHEDULER="spark.scheduler.mode=FAIR"
export
EXTRAJAVAOPTIONS="spark.executor.extraJavaOptions=-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
export
JARS="/home/hduser/jars/spark-streaming-kafka-assembly_2.11-1.6.1.jar"
export SPARKUIPORT="spark.ui.port=55555"
export SPARKDRIVERPORT="spark.driver.port=54631"
export SPARKFILESERVERPORT="spark.fileserver.port=54731"
export SPARKBLOCKMANAGERPORT="spark.blockManager.port=54832"
export SPARKKRYOSERIALIZERBUFFERMAX="spark.kryoserializer.buffer.max=512"
}
and other jar files have passed through --jars. Note that ${JAR_FILE} in my
case is built through MVN or SBT
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 1 November 2016 at 14:02, Jan Botorek <Ja...@infor.com> wrote:
> Yes, exactly.
> My (testing) run script is:
>
> spark-submit --class com.infor.skyvault.tests.LinearRegressionTest
> --master local C:\_resources\spark-1.0-SNAPSHOT.jar
> -DtrainDataPath="/path/to/model/data"
>
>
>
>
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Sent:* Tuesday, November 1, 2016 2:51 PM
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* Vinod Mangipudi <vi...@gmail.com>; user <us...@spark.apache.org>
>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> Are you submitting your job through spark-submit?
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
> On 1 November 2016 at 13:39, Jan Botorek <Ja...@infor.com> wrote:
>
> Hello,
>
> This approach unfortunately doesn’t work for job submission for me. It
> works in the shell, but not when submitted.
>
> I ensured the (only worker) node has desired directory.
>
>
>
> Neither specifying all jars as you suggested, neither using
> */path/to/jarfiles/** works.
>
>
>
> Could you verify, that using this settings you are able to submit jobs
> with according dependencies, please?
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Sent:* Tuesday, November 1, 2016 2:18 PM
> *To:* Vinod Mangipudi <vi...@gmail.com>
>
>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> you can do that as long as every node has the directory referenced.
>
>
>
> For example
>
>
>
> spark.driver.extraClassPath /home/hduser/jars/ojdbc6.jar:/
> home/hduser/jars/jconn4.jar
> spark.executor.extraClassPath /home/hduser/jars/ojdbc6.jar:/
> home/hduser/jars/jconn4.jar
>
>
>
> this will work as long as all nodes have that directory.
>
>
>
> The other alternative is to mount the shared directory as NFS mount across
> all the nodes and all the noses can read from that shared directory
>
>
>
> HTH
>
>
>
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
> On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com> wrote:
>
> unsubscribe
>
>
>
> On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com> wrote:
>
> Thank you for the reply.
>
> I am aware of the parameters used when submitting the tasks (--jars is
> working for us).
>
>
>
> But, isn’t there any way how to specify a location (directory) for jars
> „in global“ - in the spark-defaults.conf??
>
>
>
>
>
> *From:* ayan guha [mailto:guha.ayan@gmail.com]
> *Sent:* Tuesday, November 1, 2016 1:49 PM
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> There are options to specify external jars in the form of --jars,
> --driver-classpath etc depending on spark version and cluster manager..
> Please see spark documents for configuration sections and/or run spark
> submit help to see available options.
>
> On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com> wrote:
>
> Hello,
>
> I have a problem trying to add jar files to be available on classpath when
> submitting task to Spark.
>
>
>
> In my spark-defaults.conf file I have configuration:
>
> *spark.driver.extraClassPath = path/to/folder/with/jars*
>
> all jars in the folder are available in SPARK-SHELL
>
>
>
> The problem is that jars are not on the classpath for SPARK-MASTER; more
> precisely – when I submit any job that utilizes any jar from external
> folder, the* java.lang.ClassNotFoundException* is thrown.
>
> Moving all external jars into the *jars* folder solves the situation, but
> we need to keep external files separatedly.
>
>
>
> Thank you for any help
>
> Best regards,
>
> Jan
>
>
>
>
>
>
>
RE: Add jar files on classpath when submitting tasks to Spark
Posted by Jan Botorek <Ja...@infor.com>.
Yes, exactly.
My (testing) run script is:
spark-submit --class com.infor.skyvault.tests.LinearRegressionTest --master local C:\_resources\spark-1.0-SNAPSHOT.jar -DtrainDataPath="/path/to/model/data"
From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
Sent: Tuesday, November 1, 2016 2:51 PM
To: Jan Botorek <Ja...@infor.com>
Cc: Vinod Mangipudi <vi...@gmail.com>; user <us...@spark.apache.org>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
Are you submitting your job through spark-submit?
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 1 November 2016 at 13:39, Jan Botorek <Ja...@infor.com>> wrote:
Hello,
This approach unfortunately doesn’t work for job submission for me. It works in the shell, but not when submitted.
I ensured the (only worker) node has desired directory.
Neither specifying all jars as you suggested, neither using /path/to/jarfiles/* works.
Could you verify, that using this settings you are able to submit jobs with according dependencies, please?
From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com<ma...@gmail.com>]
Sent: Tuesday, November 1, 2016 2:18 PM
To: Vinod Mangipudi <vi...@gmail.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
you can do that as long as every node has the directory referenced.
For example
spark.driver.extraClassPath /home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
spark.executor.extraClassPath /home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
this will work as long as all nodes have that directory.
The other alternative is to mount the shared directory as NFS mount across all the nodes and all the noses can read from that shared directory
HTH
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com>> wrote:
unsubscribe
On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com>> wrote:
Thank you for the reply.
I am aware of the parameters used when submitting the tasks (--jars is working for us).
But, isn’t there any way how to specify a location (directory) for jars „in global“ - in the spark-defaults.conf??
From: ayan guha [mailto:guha.ayan@gmail.com<ma...@gmail.com>]
Sent: Tuesday, November 1, 2016 1:49 PM
To: Jan Botorek <Ja...@infor.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
There are options to specify external jars in the form of --jars, --driver-classpath etc depending on spark version and cluster manager.. Please see spark documents for configuration sections and/or run spark submit help to see available options.
On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com>> wrote:
Hello,
I have a problem trying to add jar files to be available on classpath when submitting task to Spark.
In my spark-defaults.conf file I have configuration:
spark.driver.extraClassPath = path/to/folder/with/jars
all jars in the folder are available in SPARK-SHELL
The problem is that jars are not on the classpath for SPARK-MASTER; more precisely – when I submit any job that utilizes any jar from external folder, the java.lang.ClassNotFoundException is thrown.
Moving all external jars into the jars folder solves the situation, but we need to keep external files separatedly.
Thank you for any help
Best regards,
Jan
Re: Add jar files on classpath when submitting tasks to Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Are you submitting your job through spark-submit?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 1 November 2016 at 13:39, Jan Botorek <Ja...@infor.com> wrote:
> Hello,
>
> This approach unfortunately doesn’t work for job submission for me. It
> works in the shell, but not when submitted.
>
> I ensured the (only worker) node has desired directory.
>
>
>
> Neither specifying all jars as you suggested, neither using
> */path/to/jarfiles/** works.
>
>
>
> Could you verify, that using this settings you are able to submit jobs
> with according dependencies, please?
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Sent:* Tuesday, November 1, 2016 2:18 PM
> *To:* Vinod Mangipudi <vi...@gmail.com>
>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> you can do that as long as every node has the directory referenced.
>
>
>
> For example
>
>
>
> spark.driver.extraClassPath /home/hduser/jars/ojdbc6.jar:/
> home/hduser/jars/jconn4.jar
> spark.executor.extraClassPath /home/hduser/jars/ojdbc6.jar:/
> home/hduser/jars/jconn4.jar
>
>
>
> this will work as long as all nodes have that directory.
>
>
>
> The other alternative is to mount the shared directory as NFS mount across
> all the nodes and all the noses can read from that shared directory
>
>
>
> HTH
>
>
>
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
> On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com> wrote:
>
> unsubscribe
>
>
>
> On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com> wrote:
>
> Thank you for the reply.
>
> I am aware of the parameters used when submitting the tasks (--jars is
> working for us).
>
>
>
> But, isn’t there any way how to specify a location (directory) for jars
> „in global“ - in the spark-defaults.conf??
>
>
>
>
>
> *From:* ayan guha [mailto:guha.ayan@gmail.com]
> *Sent:* Tuesday, November 1, 2016 1:49 PM
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> There are options to specify external jars in the form of --jars,
> --driver-classpath etc depending on spark version and cluster manager..
> Please see spark documents for configuration sections and/or run spark
> submit help to see available options.
>
> On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com> wrote:
>
> Hello,
>
> I have a problem trying to add jar files to be available on classpath when
> submitting task to Spark.
>
>
>
> In my spark-defaults.conf file I have configuration:
>
> *spark.driver.extraClassPath = path/to/folder/with/jars*
>
> all jars in the folder are available in SPARK-SHELL
>
>
>
> The problem is that jars are not on the classpath for SPARK-MASTER; more
> precisely – when I submit any job that utilizes any jar from external
> folder, the* java.lang.ClassNotFoundException* is thrown.
>
> Moving all external jars into the *jars* folder solves the situation, but
> we need to keep external files separatedly.
>
>
>
> Thank you for any help
>
> Best regards,
>
> Jan
>
>
>
>
>
RE: Add jar files on classpath when submitting tasks to Spark
Posted by Jan Botorek <Ja...@infor.com>.
Hello,
This approach unfortunately doesn’t work for job submission for me. It works in the shell, but not when submitted.
I ensured the (only worker) node has desired directory.
Neither specifying all jars as you suggested, neither using /path/to/jarfiles/* works.
Could you verify, that using this settings you are able to submit jobs with according dependencies, please?
From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
Sent: Tuesday, November 1, 2016 2:18 PM
To: Vinod Mangipudi <vi...@gmail.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
you can do that as long as every node has the directory referenced.
For example
spark.driver.extraClassPath /home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
spark.executor.extraClassPath /home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
this will work as long as all nodes have that directory.
The other alternative is to mount the shared directory as NFS mount across all the nodes and all the noses can read from that shared directory
HTH
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com>> wrote:
unsubscribe
On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com>> wrote:
Thank you for the reply.
I am aware of the parameters used when submitting the tasks (--jars is working for us).
But, isn’t there any way how to specify a location (directory) for jars „in global“ - in the spark-defaults.conf??
From: ayan guha [mailto:guha.ayan@gmail.com<ma...@gmail.com>]
Sent: Tuesday, November 1, 2016 1:49 PM
To: Jan Botorek <Ja...@infor.com>>
Cc: user <us...@spark.apache.org>>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
There are options to specify external jars in the form of --jars, --driver-classpath etc depending on spark version and cluster manager.. Please see spark documents for configuration sections and/or run spark submit help to see available options.
On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com>> wrote:
Hello,
I have a problem trying to add jar files to be available on classpath when submitting task to Spark.
In my spark-defaults.conf file I have configuration:
spark.driver.extraClassPath = path/to/folder/with/jars
all jars in the folder are available in SPARK-SHELL
The problem is that jars are not on the classpath for SPARK-MASTER; more precisely – when I submit any job that utilizes any jar from external folder, the java.lang.ClassNotFoundException is thrown.
Moving all external jars into the jars folder solves the situation, but we need to keep external files separatedly.
Thank you for any help
Best regards,
Jan
Re: Add jar files on classpath when submitting tasks to Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
you can do that as long as every node has the directory referenced.
For example
spark.driver.extraClassPath
/home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
spark.executor.extraClassPath
/home/hduser/jars/ojdbc6.jar:/home/hduser/jars/jconn4.jar
this will work as long as all nodes have that directory.
The other alternative is to mount the shared directory as NFS mount across
all the nodes and all the noses can read from that shared directory
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 1 November 2016 at 13:04, Vinod Mangipudi <vi...@gmail.com> wrote:
> unsubscribe
>
> On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com> wrote:
>
>> Thank you for the reply.
>>
>> I am aware of the parameters used when submitting the tasks (--jars is
>> working for us).
>>
>>
>>
>> But, isn’t there any way how to specify a location (directory) for jars
>> „in global“ - in the spark-defaults.conf??
>>
>>
>>
>>
>>
>> *From:* ayan guha [mailto:guha.ayan@gmail.com]
>> *Sent:* Tuesday, November 1, 2016 1:49 PM
>> *To:* Jan Botorek <Ja...@infor.com>
>> *Cc:* user <us...@spark.apache.org>
>> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>>
>>
>>
>> There are options to specify external jars in the form of --jars,
>> --driver-classpath etc depending on spark version and cluster manager..
>> Please see spark documents for configuration sections and/or run spark
>> submit help to see available options.
>>
>> On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com> wrote:
>>
>> Hello,
>>
>> I have a problem trying to add jar files to be available on classpath
>> when submitting task to Spark.
>>
>>
>>
>> In my spark-defaults.conf file I have configuration:
>>
>> *spark.driver.extraClassPath = path/to/folder/with/jars*
>>
>> all jars in the folder are available in SPARK-SHELL
>>
>>
>>
>> The problem is that jars are not on the classpath for SPARK-MASTER; more
>> precisely – when I submit any job that utilizes any jar from external
>> folder, the* java.lang.ClassNotFoundException* is thrown.
>>
>> Moving all external jars into the *jars* folder solves the situation,
>> but we need to keep external files separatedly.
>>
>>
>>
>> Thank you for any help
>>
>> Best regards,
>>
>> Jan
>>
>>
>
Re: Add jar files on classpath when submitting tasks to Spark
Posted by Vinod Mangipudi <vi...@gmail.com>.
unsubscribe
On Tue, Nov 1, 2016 at 8:56 AM, Jan Botorek <Ja...@infor.com> wrote:
> Thank you for the reply.
>
> I am aware of the parameters used when submitting the tasks (--jars is
> working for us).
>
>
>
> But, isn’t there any way how to specify a location (directory) for jars
> „in global“ - in the spark-defaults.conf??
>
>
>
>
>
> *From:* ayan guha [mailto:guha.ayan@gmail.com]
> *Sent:* Tuesday, November 1, 2016 1:49 PM
> *To:* Jan Botorek <Ja...@infor.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Add jar files on classpath when submitting tasks to Spark
>
>
>
> There are options to specify external jars in the form of --jars,
> --driver-classpath etc depending on spark version and cluster manager..
> Please see spark documents for configuration sections and/or run spark
> submit help to see available options.
>
> On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com> wrote:
>
> Hello,
>
> I have a problem trying to add jar files to be available on classpath when
> submitting task to Spark.
>
>
>
> In my spark-defaults.conf file I have configuration:
>
> *spark.driver.extraClassPath = path/to/folder/with/jars*
>
> all jars in the folder are available in SPARK-SHELL
>
>
>
> The problem is that jars are not on the classpath for SPARK-MASTER; more
> precisely – when I submit any job that utilizes any jar from external
> folder, the* java.lang.ClassNotFoundException* is thrown.
>
> Moving all external jars into the *jars* folder solves the situation, but
> we need to keep external files separatedly.
>
>
>
> Thank you for any help
>
> Best regards,
>
> Jan
>
>
RE: Add jar files on classpath when submitting tasks to Spark
Posted by Jan Botorek <Ja...@infor.com>.
Thank you for the reply.
I am aware of the parameters used when submitting the tasks (--jars is working for us).
But, isn’t there any way how to specify a location (directory) for jars „in global“ - in the spark-defaults.conf??
From: ayan guha [mailto:guha.ayan@gmail.com]
Sent: Tuesday, November 1, 2016 1:49 PM
To: Jan Botorek <Ja...@infor.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Add jar files on classpath when submitting tasks to Spark
There are options to specify external jars in the form of --jars, --driver-classpath etc depending on spark version and cluster manager.. Please see spark documents for configuration sections and/or run spark submit help to see available options.
On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com>> wrote:
Hello,
I have a problem trying to add jar files to be available on classpath when submitting task to Spark.
In my spark-defaults.conf file I have configuration:
spark.driver.extraClassPath = path/to/folder/with/jars
all jars in the folder are available in SPARK-SHELL
The problem is that jars are not on the classpath for SPARK-MASTER; more precisely – when I submit any job that utilizes any jar from external folder, the java.lang.ClassNotFoundException is thrown.
Moving all external jars into the jars folder solves the situation, but we need to keep external files separatedly.
Thank you for any help
Best regards,
Jan
Re: Add jar files on classpath when submitting tasks to Spark
Posted by ayan guha <gu...@gmail.com>.
There are options to specify external jars in the form of --jars,
--driver-classpath etc depending on spark version and cluster manager..
Please see spark documents for configuration sections and/or run spark
submit help to see available options.
On 1 Nov 2016 23:13, "Jan Botorek" <Ja...@infor.com> wrote:
> Hello,
>
> I have a problem trying to add jar files to be available on classpath when
> submitting task to Spark.
>
>
>
> In my spark-defaults.conf file I have configuration:
>
> *spark.driver.extraClassPath = path/to/folder/with/jars*
>
> all jars in the folder are available in SPARK-SHELL
>
>
>
> The problem is that jars are not on the classpath for SPARK-MASTER; more
> precisely – when I submit any job that utilizes any jar from external
> folder, the* java.lang.ClassNotFoundException* is thrown.
>
> Moving all external jars into the *jars* folder solves the situation, but
> we need to keep external files separatedly.
>
>
>
> Thank you for any help
>
> Best regards,
>
> Jan
>