You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ray <ra...@cisco.com> on 2018/03/19 08:27:37 UTC

DataFrame support for Apache Spark 1.6

I'm trying to save spark dataframe to Ignite 2.4 using Apache Spark 1.6.
But it failed, with the following error
Exception in thread "main" java.util.ServiceConfigurationError:
org.apache.spark.sql.sources.DataSourceRegister: Provider
org.apache.ignite.spark.impl.IgniteRelationProvider could not be
instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:232)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:42)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$class.filter(TraversableLike.scala:263)
	at scala.collection.AbstractTraversable.filter(Traversable.scala:105)
	at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:59)
	at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
	at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:244)
	at IgniteDataFrameWriteExample$.main(IgniteDataFrameWriteExample.scala:40)
	at IgniteDataFrameWriteExample.main(IgniteDataFrameWriteExample.scala)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
	at java.lang.Class.getConstructor0(Class.java:3075)
	at java.lang.Class.newInstance(Class.java:412)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	... 16 more
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.internal.Logging
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 33 more

But it works fine under Spark 2.2.
So I'm wondering will Spark dataframe feature supports Spark 1.6 in the
future?
I can't upgrade to Spark 2.2 because Cloudera won't upgrade.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: DataFrame support for Apache Spark 1.6

Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Ray.

Currently, There is no plans to support Spark 1.6 in Ignite.
I doubt if it can be done without significat changes in existing code base.

Anyway, you can create a ticket [1].
And I will try to look what can be done.

[1] https://issues.apache.org/jira/browse/IGNITE

В Пн, 19/03/2018 в 01:27 -0700, Ray пишет:
> I'm trying to save spark dataframe to Ignite 2.4 using Apache Spark 1.6.
> But it failed, with the following error
> Exception in thread "main" java.util.ServiceConfigurationError:
> org.apache.spark.sql.sources.DataSourceRegister: Provider
> org.apache.ignite.spark.impl.IgniteRelationProvider could not be
> instantiated
> 	at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> 	at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> 	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> 	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> 	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> 	at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:42)
> 	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> 	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> 	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> 	at scala.collection.TraversableLike$class.filter(TraversableLike.scala:263)
> 	at scala.collection.AbstractTraversable.filter(Traversable.scala:105)
> 	at
> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:59)
> 	at
> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102)
> 	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
> 	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
> 	at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:244)
> 	at IgniteDataFrameWriteExample$.main(IgniteDataFrameWriteExample.scala:40)
> 	at IgniteDataFrameWriteExample.main(IgniteDataFrameWriteExample.scala)
> Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging
> 	at java.lang.ClassLoader.defineClass1(Native Method)
> 	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> 	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> 	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> 	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	at java.lang.Class.getDeclaredConstructors0(Native Method)
> 	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> 	at java.lang.Class.getConstructor0(Class.java:3075)
> 	at java.lang.Class.newInstance(Class.java:412)
> 	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> 	... 16 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.internal.Logging
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	... 33 more
> 
> But it works fine under Spark 2.2.
> So I'm wondering will Spark dataframe feature supports Spark 1.6 in the
> future?
> I can't upgrade to Spark 2.2 because Cloudera won't upgrade.
> 
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/