Re: JDBC DF using DB2

[Ted Yu <yu...@gmail.com>]

bq. is to modify compute_classpath.sh on all worker nodes to include your
driver JARs.

Please follow the above advice.

Cheers

On Mon, Mar 23, 2015 at 12:34 PM, Jack Arenas <j...@ckarenas.com> wrote:

> Hi Team,
>
>
>
> I’m trying to create a DF using jdbc as detailed here
> <https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#jdbc-to-other-databases> –
> I’m currently using DB2 v9.7.0.6 and I’ve tried to use the db2jcc.jar and
> db2jcc_license_cu.jar combo, and while it works in --master local using the
> command below, I get some strange behavior in --master yarn-client. Here is
> the command:
>
>
>
> *val df = sql.load("jdbc", Map("url" ->
> "jdbc:db2://<host>:<port>/<db>:currentSchema=<schema>;user=<user>;password=<password>;",
> "driver" -> "com.ibm.db2.jcc.DB2Driver", "dbtable" -> "<table>"))*
>
>
>
> It seems to also be working on yarn-client because once executed I get the
> following log:
>
> *df: org.apache.spark.sql.DataFrame = [DATE_FIELD: date, INT_FIELD: int,
> DOUBLE_FIELD: double]*
>
>
>
> Which indicates me that Spark was able to connect to the DB. But once I
> run *df.count() *or *df.take(5).foreach(println)* in order to operate on
> the data and get a result, I get back a ‘*No suitable driver found*’
> exception, which makes me think the driver wasn’t shipped with the spark
> job.
>
>
>
> I’ve tried using *--driver-class-path, --jars, SPARK_CLASSPATH* to add
> the jars to the spark job. I also have the jars in my*$CLASSPATH* and
> *$HADOOP_CLASSPATH*.
>
>
>
> I also saw this in the trouble shooting section, but quite frankly I’m not
> sure what primordial class loader it’s talking about:
>
>
>
> The JDBC driver class must be visible to the primordial class loader on
> the client session and on all executors. This is because Java’s
> DriverManager class does a security check that results in it ignoring all
> drivers not visible to the primordial class loader when one goes to open a
> connection. One convenient way to do this is to modify compute_classpath.sh
> on all worker nodes to include your driver JARs.
>
>
> Any advice is welcome!
>
>
>
> Thanks,
>
> Jack
>
>
>
>
>
>
>