You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@peridale.co.uk> on 2016/02/15 00:35:27 UTC
Passing multiple jar files to spark-shell
Hi,
Is there anyway one can pass multiple --driver-class-path and multiple -jars
to spark shell.
For example something as below with two jar files entries for Oracle
(ojdbc6.jar) and Sybase IQ (jcoon4,jar)
spark-shell --master spark://50.140.197.217:7077 --driver-class-path
/home/hduser/jars/jconn4.jar --driver-class-path
/home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
/home/hduser/jars/ojdbc6.jar
This works for one jar file only and you need to add the jar file to both
--driver-class-path and -jars. I have not managed to work for more than one
type of JAR file
Thanks,
Dr Mich Talebzadeh
LinkedIn
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABU
rV8Pw>
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUr
V8Pw
<http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Technology Ltd, its subsidiaries nor their
employees accept any responsibility.
Re: Passing multiple jar files to spark-shell
Posted by Sathish Kumaran Vairavelu <vs...@gmail.com>.
--jars takes comma separated values.
On Sun, Feb 14, 2016 at 5:35 PM Mich Talebzadeh <mi...@peridale.co.uk> wrote:
> Hi,
>
>
>
> Is there anyway one can pass multiple --driver-class-path and multiple
> –jars to spark shell.
>
>
>
> For example something as below with two jar files entries for Oracle
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --driver-class-path
> /home/hduser/jars/jconn4.jar --driver-class-path
> /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
> /home/hduser/jars/ojdbc6.jar
>
>
>
>
>
> This works for one jar file only and you need to add the jar file to both --driver-class-path
> and –jars. I have not managed to work for more than one type of JAR file
>
>
>
> Thanks,
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
Re: Passing multiple jar files to spark-shell
Posted by Mich Talebzadeh <mi...@cloudtechnologypartners.co.uk>.
Thanks Ted. I will have a look.
regards
On 15/02/2016 14:34, Ted Yu wrote:
> Mich:
> You can pass jars for driver using:
>
> spark.driver.extraClassPath
>
> Cheers
>
> On Mon, Feb 15, 2016 at 1:05 AM, Mich Talebzadeh <mi...@peridale.co.uk> wrote:
>
> Thanks Deng. Unfortunately it seems that it looks for driver-class-path as well L
>
> For example with -jars alone I get
>
> spark-shell --master spark://50.140.197.217:7077 [1] --jars /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar
>
> s: org.apache.spark.sql.DataFrame = [AMOUNT_SOLD: decimal(10,0), TIME_ID: timestamp, CHANNEL_ID: bigint]
>
> warning: there were 1 deprecation warning(s); re-run with -deprecation for details
>
> JAVA.SQL.SQLEXCEPTION: NO SUITABLE DRIVER FOUND FOR JDBC:ORACLE:THIN:@RHES564:1521:MYDB
>
> Dr Mich Talebzadeh
>
> LinkedIn _ https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw [2]_
>
> http://talebzadehmich.wordpress.com [3]
>
> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Technology Ltd, its subsidiaries nor their employees accept any responsibility.
>
> FROM: odeaching@gmail.com [mailto:odeaching@gmail.com] ON BEHALF OF Deng Ching-Mallete
> SENT: 15 February 2016 03:27
> TO: Mich Talebzadeh <mi...@peridale.co.uk>
> CC: user <us...@spark.apache.org>
> SUBJECT: Re: Passing multiple jar files to spark-shell
>
> Hi Mich,
>
> For the --jars parameter, just pass in the jars as comma-delimited. As for the --driver-class-path, make it colon-delimited -- similar to how you set multiple paths for an environment variable (e.g. --driver-class-path /home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're already passing the jar files via the --jars param, you shouldn't need to set them in the driver-class-path though, since they should already be automatically added to the classpath.
>
> HTH,
>
> Deng
>
> On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <mi...@peridale.co.uk> wrote:
>
> Hi,
>
> Is there anyway one can pass multiple --driver-class-path and multiple -jars to spark shell.
>
> For example something as below with two jar files entries for Oracle (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
> spark-shell --master spark://50.140.197.217:7077 [1] --driver-class-path /home/hduser/jars/jconn4.jar --driver-class-path /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars /home/hduser/jars/ojdbc6.jar
>
> This works for one jar file only and you need to add the jar file to both --driver-class-path and -jars. I have not managed to work for more than one type of JAR file
>
> Thanks,
>
> Dr Mich Talebzadeh
>
> LinkedIn _ https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw [2]_
>
> http://talebzadehmich.wordpress.com [3]
>
> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Technology Ltd, its subsidiaries nor their employees accept any responsibility.
--
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
NOTE: The information in this email is proprietary and confidential.
This message is for the designated recipient only, if you are not the
intended recipient, you should destroy it immediately. Any information
in this message shall not be understood as given or endorsed by Cloud
Technology Partners Ltd, its subsidiaries or their employees, unless
expressly so stated. It is the responsibility of the recipient to ensure
that this email is virus free, therefore neither Cloud Technology
partners Ltd, its subsidiaries nor their employees accept any
responsibility.
Links:
------
[1] http://50.140.197.217:7077
[2]
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
[3] http://talebzadehmich.wordpress.com/
Re: Passing multiple jar files to spark-shell
Posted by Ted Yu <yu...@gmail.com>.
Mich:
You can pass jars for driver using:
spark.driver.extraClassPath
Cheers
On Mon, Feb 15, 2016 at 1:05 AM, Mich Talebzadeh <mi...@peridale.co.uk>
wrote:
> Thanks Deng. Unfortunately it seems that it looks for driver-class-path as
> well L
>
>
>
> For example with –jars alone I get
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --jars
> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar
>
>
>
>
>
> s: org.apache.spark.sql.DataFrame = [AMOUNT_SOLD: decimal(10,0), TIME_ID:
> timestamp, CHANNEL_ID: bigint]
>
> warning: there were 1 deprecation warning(s); re-run with -deprecation for
> details
>
> *java.sql.SQLException: No suitable driver found for
> jdbc:oracle:thin:@rhes564:1521:mydb*
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
> *From:* odeaching@gmail.com [mailto:odeaching@gmail.com] *On Behalf Of *Deng
> Ching-Mallete
> *Sent:* 15 February 2016 03:27
> *To:* Mich Talebzadeh <mi...@peridale.co.uk>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Passing multiple jar files to spark-shell
>
>
>
> Hi Mich,
>
>
>
> For the --jars parameter, just pass in the jars as comma-delimited. As for
> the --driver-class-path, make it colon-delimited -- similar to how you set
> multiple paths for an environment variable (e.g. --driver-class-path
> /home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're
> already passing the jar files via the --jars param, you shouldn't need to
> set them in the driver-class-path though, since they should already be
> automatically added to the classpath.
>
>
>
> HTH,
>
> Deng
>
>
>
> On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <mi...@peridale.co.uk>
> wrote:
>
> Hi,
>
>
>
> Is there anyway one can pass multiple --driver-class-path and multiple
> –jars to spark shell.
>
>
>
> For example something as below with two jar files entries for Oracle
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --driver-class-path
> /home/hduser/jars/jconn4.jar --driver-class-path
> /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
> /home/hduser/jars/ojdbc6.jar
>
>
>
>
>
> This works for one jar file only and you need to add the jar file to both --driver-class-path
> and –jars. I have not managed to work for more than one type of JAR file
>
>
>
> Thanks,
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>
>
>
>
>
RE: Passing multiple jar files to spark-shell
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Thanks Deng. Unfortunately it seems that it looks for driver-class-path as well :(
For example with –jars alone I get
spark-shell --master spark://50.140.197.217:7077 --jars /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar
s: org.apache.spark.sql.DataFrame = [AMOUNT_SOLD: decimal(10,0), TIME_ID: timestamp, CHANNEL_ID: bigint]
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
java.sql.SQLException: No suitable driver found for jdbc:oracle:thin:@rhes564:1521:mydb
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Technology Ltd, its subsidiaries nor their employees accept any responsibility.
From: odeaching@gmail.com [mailto:odeaching@gmail.com] On Behalf Of Deng Ching-Mallete
Sent: 15 February 2016 03:27
To: Mich Talebzadeh <mi...@peridale.co.uk>
Cc: user <us...@spark.apache.org>
Subject: Re: Passing multiple jar files to spark-shell
Hi Mich,
For the --jars parameter, just pass in the jars as comma-delimited. As for the --driver-class-path, make it colon-delimited -- similar to how you set multiple paths for an environment variable (e.g. --driver-class-path /home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're already passing the jar files via the --jars param, you shouldn't need to set them in the driver-class-path though, since they should already be automatically added to the classpath.
HTH,
Deng
On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <mich@peridale.co.uk <ma...@peridale.co.uk> > wrote:
Hi,
Is there anyway one can pass multiple --driver-class-path and multiple –jars to spark shell.
For example something as below with two jar files entries for Oracle (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
spark-shell --master spark://50.140.197.217:7077 <http://50.140.197.217:7077> --driver-class-path /home/hduser/jars/jconn4.jar --driver-class-path /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars /home/hduser/jars/ojdbc6.jar
This works for one jar file only and you need to add the jar file to both --driver-class-path and –jars. I have not managed to work for more than one type of JAR file
Thanks,
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Technology Ltd, its subsidiaries nor their employees accept any responsibility.
Re: Passing multiple jar files to spark-shell
Posted by Deng Ching-Mallete <oc...@apache.org>.
Hi Mich,
For the --jars parameter, just pass in the jars as comma-delimited. As for
the --driver-class-path, make it colon-delimited -- similar to how you set
multiple paths for an environment variable (e.g. --driver-class-path
/home/hduser/jars/jconn4.jar:/home/hduse/jars/ojdbc6.jar). But if you're
already passing the jar files via the --jars param, you shouldn't need to
set them in the driver-class-path though, since they should already be
automatically added to the classpath.
HTH,
Deng
On Mon, Feb 15, 2016 at 7:35 AM, Mich Talebzadeh <mi...@peridale.co.uk>
wrote:
> Hi,
>
>
>
> Is there anyway one can pass multiple --driver-class-path and multiple
> –jars to spark shell.
>
>
>
> For example something as below with two jar files entries for Oracle
> (ojdbc6.jar) and Sybase IQ (jcoon4,jar)
>
>
>
> spark-shell --master spark://50.140.197.217:7077 --driver-class-path
> /home/hduser/jars/jconn4.jar --driver-class-path
> /home/hduser/jars/ojdbc6.jar --jars /home/hduser/jars/jconn4.jar --jars
> /home/hduser/jars/ojdbc6.jar
>
>
>
>
>
> This works for one jar file only and you need to add the jar file to both --driver-class-path
> and –jars. I have not managed to work for more than one type of JAR file
>
>
>
> Thanks,
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Technology Ltd, its subsidiaries nor their
> employees accept any responsibility.
>
>
>
>
>