You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Devl Devel <de...@gmail.com> on 2015/04/11 13:00:41 UTC
Apache Ignite Integration with Spark RDDs
Hi Ingiters
Spark has an RDD interface which lets you plug in various back-ends like
HDFS and JDBC etc. I think it would be a great idea if the community could
leverage Ignites in-memory file system to provide blazing fast analytics
using both ingite and spark.
Here is the interface :
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala
The idea would be to compute splits and partitions from ignitefs and use
these in an RDD, an example is:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
Would like to know whether this is feasible, as it would be a useful
addition to both communities and help complement other implementations for
tachyon for example.
Cheers
Devl
Re: Apache Ignite Integration with Spark RDDs
Posted by Konstantin Boudnik <co...@apache.org>.
I think it'd be great, as it will open even deeper level of the integration
between the two projects!
For the reference, there's already a JIRA opened for this feature
https://issues.apache.org/jira/browse/IGNITE-389
Cos
On Sat, Apr 11, 2015 at 12:00PM, Devl Devel wrote:
> Hi Ingiters
>
> Spark has an RDD interface which lets you plug in various back-ends like
> HDFS and JDBC etc. I think it would be a great idea if the community could
> leverage Ignites in-memory file system to provide blazing fast analytics
> using both ingite and spark.
>
> Here is the interface :
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala
>
> The idea would be to compute splits and partitions from ignitefs and use
> these in an RDD, an example is:
>
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
>
> Would like to know whether this is feasible, as it would be a useful
> addition to both communities and help complement other implementations for
> tachyon for example.
>
> Cheers
> Devl