You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Guy Needham <gu...@gmail.com> on 2017/11/19 11:25:43 UTC
MXNet on Flink
MXNet (incubating), https://mxnet.incubator.apache.org/, is a pretty cool
deep learning framework. It allows the user to develop across languages and
to share the model model objects between languages. This is useful as it
allows us to develop models in Python, then export the model to a file
which for example a Spark job can pick up and continue to train the model
in a distributed setting. This flexibility and additionally the high
performance of the library differentiate it from other frameworks.
In Spark, the driver process contains a parameter server allowing worker
processes to communicate information on the model training asynchronously
which is a common pattern in the distribution of model training. Source can
be found here:
https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark
I think it would be useful to be able to use MXNet with Flink in the same
way one can use MXNet with Spark to train a model in a distributed setting
on a large stream of data.
Is anyone else interested in this?
Re: MXNet on Flink
Posted by Sugandha Amatya <su...@gmail.com>.
This ia dope.
Sent from Astro <https://www.helloastro.com> for Android
On 11/19/17 at 5:10 PM, Guy wrote:
MXNet (incubating), https://mxnet.incubator.apache.org/, is a pretty cool
deep learning framework. It allows the user to develop across languages and
to share the model model objects between languages. This is useful as it
allows us to develop models in Python, then export the model to a file
which for example a Spark job can pick up and continue to train the model
in a distributed setting. This flexibility and additionally the high
performance of the library differentiate it from other frameworks.
In Spark, the driver process contains a parameter server allowing worker
processes to communicate information on the model training asynchronously
which is a common pattern in the distribution of model training. Source can
be found here:
https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark
I think it would be useful to be able to use MXNet with Flink in the same
way one can use MXNet with Spark to train a model in a distributed setting
on a large stream of data.
Is anyone else interested in this?