You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by Balakumar <ba...@citrix.com> on 2019/01/23 05:49:08 UTC

PySpark: Failed to find data source: ignite

Hi,

I'm trying to put parquet in to Ignite table, but getting the below error.

java.lang.ClassNotFoundException: Failed to find data source: ignite. Please
find packages at http://spark.apache.org/third-party-projects.html

Spark: 2.3.1
Trying with PySpark
I have copied the ignite-spark from optional to lib folder, and followed
classpath inclusion for Ignite for spark-env.sh.

Here is the code



Thanks,
Bala



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: PySpark: Failed to find data source: ignite

Posted by Balakumar <ba...@citrix.com>.

Hi Stephan,

Thanks for the reply.

I have actually added following lib settings in spark-defaults.conf

spark.driver.extraClassPath /opt/ignite/libs/*.jar
spark.driver.extraClassPath /opt/ignite/libs/ignite-indexing/*.jar
spark.driver.extraClassPath /opt/ignite/libs/ignite-spark/*.jar
spark.driver.extraClassPath /opt/ignite/libs/ignite-spring/*.jar

spark.executor.extraClassPath /opt/ignite/libs/*.jar
spark.executor.extraClassPath /opt/ignite/libs/ignite-indexing/*.jar
spark.executor.extraClassPath /opt/ignite/libs/ignite-spark/*.jar
spark.executor.extraClassPath /opt/ignite/libs/ignite-spring/*.jar

spark.driver.extraLibraryPath /opt/ignite/libs
spark.driver.extraLibraryPath /opt/ignite/libs/ignite-indexing
spark.driver.extraLibraryPath /opt/ignite/libs/ignite-spark
spark.driver.extraLibraryPath /opt/ignite/libs/ignite-spring

spark.executor.extraLibraryPath /opt/ignite/libs
spark.executor.extraLibraryPath /opt/ignite/libs/ignite-indexing
spark.executor.extraLibraryPath /opt/ignite/libs/ignite-spark
spark.executor.extraLibraryPath /opt/ignite/libs/ignite-spring

I got the way you initialized in command line, I actually initialize the
context/session through application (python script).

I have tried the same thing while creating spark session also (passing as
configs)

Is there any other way to get the ignite format working for Spark.

Thanks,
Bala



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: PySpark: Failed to find data source: ignite

Posted by Stephen Darlington <st...@gridgain.com>.

You don’t say what your full CLASSPATH is but you’re clearly missing something. Here’s how I did it:

https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba

Regards,
Stephen

> On 23 Jan 2019, at 05:49, Balakumar <ba...@citrix.com> wrote:
> 
> Hi,
> 
> I'm trying to put parquet in to Ignite table, but getting the below error.
> 
> java.lang.ClassNotFoundException: Failed to find data source: ignite. Please
> find packages at http://spark.apache.org/third-party-projects.html
> 
> Spark: 2.3.1
> Trying with PySpark
> I have copied the ignite-spark from optional to lib folder, and followed
> classpath inclusion for Ignite for spark-env.sh.
> 
> Here is the code
> 
> 
> 
> Thanks,
> Bala
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/