Ignite and spark for deep learning

[mehdi sey <se...@gmail.com>]

Hi. two platforms (spark and ignite) use in memory for computing. instead of
loading data into ignite catch we also can loading data to spark memory and
catch it on spark node. if we could do  this (catching on spark node) why we
load data to ignite catch?. loading to ignite catch have benefit only for
sharing rdd between spark jobs ad indexing query index.i want to integrate
spark and ignite for deep learning platform. I want to use DL4J (deep
learning 4 java) as platform for deep learning. I want to use dl4j in spark
node and integrate spark node with ignite. Is there any speed up in this
idea,?if i want to use ignite i can use ignite only for cache data for
spark?or i can use spark and ignite as an engeen process simultaneously?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite and spark for deep learning

[Mehdi Seydali <se...@gmail.com>]

I have in doubt. In last mail you said that dl4j over ignite have no speed
up and you say now maybe develope in future.is it worthwhile for doing it?

On Friday, January 11, 2019, zaleslaw <za...@gmail.com> wrote:

> Currently we have no integration with DL4j in ignite ML and have no such
> plans in roadmap.
> The TF is the best platform for the Deep Learning now and we spend time on
> integration with TF first of all.
>
> Maybe, in future, direct integration between DL4j and ignite will be added.
> But it costst a hundred hours of development, of course.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite and spark for deep learning

[zaleslaw <za...@gmail.com>]

Currently we have no integration with DL4j in ignite ML and have no such
plans in roadmap.
The TF is the best platform for the Deep Learning now and we spend time on
integration with TF first of all.

Maybe, in future, direct integration between DL4j and ignite will be added.
But it costst a hundred hours of development, of course.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite and spark for deep learning

[Mehdi Seydali <se...@gmail.com>]

I completely agree with you. I discuss many time with our team that this
integration have not any gain about speed up and i have your idea about
caching in ignite because in deep learning we have nothing to share in job
because every job independently works on it's portion of data. In your
opinion one idea can be ignite+ignite ml+ dl4j +spark? What benefit we can
achive in this integration. ?ignite ml cant used for deep independently?

On Wednesday, January 9, 2019, zaleslaw <za...@gmail.com> wrote:

> Dear Mehdi Sey
>
> Yes, both platforms are used for in-memory computing, but they have
> different APIs and history of feature creation and different ways of
> integration with famous DL frameworks (like DL4j and TensorFlow).
>
> From my point of view, you have no speed up in Ignite + Spark + DL4j
> integration.
>
> Caching data in Ignite as a backend for RDD and dataframes first of all is
> acceleration of business logic based on SQL queries. Not the same for ML
> frameworks.
>
> We have no proof, that usage Ignite as a backend could speed up DL4j or
> MLlib algorithms.
>
> Moreover, to avoid this, we wrote own ML library which is more better than
> MLlib and runs natively on Ignite.
>
> In my opinon, you should choose Ignite + Ignite ML + TF integration or
> Spark
> + DL4j to solve your Data Science task (where you need neural networks).
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite and spark for deep learning

[zaleslaw <za...@gmail.com>]

Dear Mehdi Sey

Yes, both platforms are used for in-memory computing, but they have
different APIs and history of feature creation and different ways of
integration with famous DL frameworks (like DL4j and TensorFlow).

From my point of view, you have no speed up in Ignite + Spark + DL4j
integration.

Caching data in Ignite as a backend for RDD and dataframes first of all is
acceleration of business logic based on SQL queries. Not the same for ML
frameworks. 

We have no proof, that usage Ignite as a backend could speed up DL4j or
MLlib algorithms.

Moreover, to avoid this, we wrote own ML library which is more better than
MLlib and runs natively on Ignite.

In my opinon, you should choose Ignite + Ignite ML + TF integration or Spark
+ DL4j to solve your Data Science task (where you need neural networks).





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/