You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satish chandra j <js...@gmail.com> on 2015/11/13 16:14:15 UTC
No suitable drivers found for postgresql
HI All,
Currently using Spark 1.4.1, my Spark job has to fetche data from
PostgreSQL database using JdbcRDD
I am submitting my spark job using --jars to pass PostgreSQL JDBC driver
but still getting error as mentioned below:
"java.sql.SQLException: No suitable driver found for PostgreSQL JDBC"
when the same is given through Spark Shell it is working fine
In several blogs it is mentioned that it is fixed in Spark 1.4.1 by just
passing JDBC Driver through --jars option but still i am stuck
I have tried below options:
1. SPARK_CLASSPATH= /path/postgresql.jar in
spark/conf/spark-defaults.conf
2. -driver-class-path /path/postgresql.jar and -conf
spark.executor.extraClassPath = /path/postgreSQL.jar
3. --jars /path/postgreSQL,jar
4. Currently trying to add SPARK_CLASSPATH in file
"compute_classpath.sh" for each node of cluster
Please let me know if any inputs on the same to proceed further
Regards
Satish Chandra
Re: No suitable drivers found for postgresql
Posted by James Nowell <jn...@gmail.com>.
The --jars flag should be available for PySpark as well (I could be wrong,
I've only used Spark 1.4 onward). Take, for example, the command I'm using
to stark a PySpark shell for a Jupyter Notebook:
"--jars hdfs://{our namenode}/tmp/postgresql-9.4-1204.jdbc42.jar
--driver-class-path
/usr/local/share/jupyter/kernels/postgres/postgresql-9.4-1204.jdbc42.jar
--executor-memory 1G --total-executor-cores 15 pyspark-shell"
James
On Fri, Nov 13, 2015 at 12:12 PM Krishna Sangeeth KS <
kskrishnasangeeth@gmail.com> wrote:
>
> Hi,
>
> I have been trying to do this today at work with impala as the data source
> . I have been getting the same error as well.
>
> I am using PySpark APIs with Spark 1.3 version and I was wondering if
> there is any workaround for Pyspark. I don't think we can use --jars option
> in PySpark.
>
> Cheers,
> Sangeeth
>
>
> On Fri, Nov 13, 2015, 9:04 PM James Nowell <jn...@gmail.com> wrote:
>
>> I recently had this same issue. Though I didn't find the cause, I was
>> able to work around it by loading the JAR into hdfs. Once in HDFS, I used
>> the --jars flag with the full hdfs path: --jars hdfs://{our
>> namenode}/tmp/postgresql-9.4-1204-jdbc42.jar
>>
>> James
>>
>> On Fri, Nov 13, 2015 at 10:14 AM satish chandra j <
>> jsatishchandra@gmail.com> wrote:
>>
>>> HI All,
>>> Currently using Spark 1.4.1, my Spark job has to fetche data from
>>> PostgreSQL database using JdbcRDD
>>> I am submitting my spark job using --jars to pass PostgreSQL JDBC driver
>>> but still getting error as mentioned below:
>>>
>>> "java.sql.SQLException: No suitable driver found for PostgreSQL JDBC"
>>>
>>> when the same is given through Spark Shell it is working fine
>>>
>>> In several blogs it is mentioned that it is fixed in Spark 1.4.1 by just
>>> passing JDBC Driver through --jars option but still i am stuck
>>>
>>> I have tried below options:
>>>
>>>
>>> 1. SPARK_CLASSPATH= /path/postgresql.jar in
>>> spark/conf/spark-defaults.conf
>>>
>>> 2. -driver-class-path /path/postgresql.jar and -conf
>>> spark.executor.extraClassPath = /path/postgreSQL.jar
>>>
>>> 3. --jars /path/postgreSQL,jar
>>>
>>> 4. Currently trying to add SPARK_CLASSPATH in file
>>> "compute_classpath.sh" for each node of cluster
>>>
>>> Please let me know if any inputs on the same to proceed further
>>>
>>> Regards
>>> Satish Chandra
>>>
>>
Re: No suitable drivers found for postgresql
Posted by Krishna Sangeeth KS <ks...@gmail.com>.
Hi,
I have been trying to do this today at work with impala as the data source
. I have been getting the same error as well.
I am using PySpark APIs with Spark 1.3 version and I was wondering if
there is any workaround for Pyspark. I don't think we can use --jars option
in PySpark.
Cheers,
Sangeeth
On Fri, Nov 13, 2015, 9:04 PM James Nowell <jn...@gmail.com> wrote:
> I recently had this same issue. Though I didn't find the cause, I was able
> to work around it by loading the JAR into hdfs. Once in HDFS, I used the
> --jars flag with the full hdfs path: --jars hdfs://{our
> namenode}/tmp/postgresql-9.4-1204-jdbc42.jar
>
> James
>
> On Fri, Nov 13, 2015 at 10:14 AM satish chandra j <
> jsatishchandra@gmail.com> wrote:
>
>> HI All,
>> Currently using Spark 1.4.1, my Spark job has to fetche data from
>> PostgreSQL database using JdbcRDD
>> I am submitting my spark job using --jars to pass PostgreSQL JDBC driver
>> but still getting error as mentioned below:
>>
>> "java.sql.SQLException: No suitable driver found for PostgreSQL JDBC"
>>
>> when the same is given through Spark Shell it is working fine
>>
>> In several blogs it is mentioned that it is fixed in Spark 1.4.1 by just
>> passing JDBC Driver through --jars option but still i am stuck
>>
>> I have tried below options:
>>
>>
>> 1. SPARK_CLASSPATH= /path/postgresql.jar in
>> spark/conf/spark-defaults.conf
>>
>> 2. -driver-class-path /path/postgresql.jar and -conf
>> spark.executor.extraClassPath = /path/postgreSQL.jar
>>
>> 3. --jars /path/postgreSQL,jar
>>
>> 4. Currently trying to add SPARK_CLASSPATH in file
>> "compute_classpath.sh" for each node of cluster
>>
>> Please let me know if any inputs on the same to proceed further
>>
>> Regards
>> Satish Chandra
>>
>
Re: No suitable drivers found for postgresql
Posted by James Nowell <jn...@gmail.com>.
I recently had this same issue. Though I didn't find the cause, I was able
to work around it by loading the JAR into hdfs. Once in HDFS, I used the
--jars flag with the full hdfs path: --jars hdfs://{our
namenode}/tmp/postgresql-9.4-1204-jdbc42.jar
James
On Fri, Nov 13, 2015 at 10:14 AM satish chandra j <js...@gmail.com>
wrote:
> HI All,
> Currently using Spark 1.4.1, my Spark job has to fetche data from
> PostgreSQL database using JdbcRDD
> I am submitting my spark job using --jars to pass PostgreSQL JDBC driver
> but still getting error as mentioned below:
>
> "java.sql.SQLException: No suitable driver found for PostgreSQL JDBC"
>
> when the same is given through Spark Shell it is working fine
>
> In several blogs it is mentioned that it is fixed in Spark 1.4.1 by just
> passing JDBC Driver through --jars option but still i am stuck
>
> I have tried below options:
>
>
> 1. SPARK_CLASSPATH= /path/postgresql.jar in
> spark/conf/spark-defaults.conf
>
> 2. -driver-class-path /path/postgresql.jar and -conf
> spark.executor.extraClassPath = /path/postgreSQL.jar
>
> 3. --jars /path/postgreSQL,jar
>
> 4. Currently trying to add SPARK_CLASSPATH in file
> "compute_classpath.sh" for each node of cluster
>
> Please let me know if any inputs on the same to proceed further
>
> Regards
> Satish Chandra
>