You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Henri Yandell <ba...@apache.org> on 2017/02/21 07:54:17 UTC
ZeroMQ dependency
How tied is MXNet to ZeroMQ?
My notes are that ps-lite depends on it.
Options I can see here are:
1) Discuss on general@incubator and determine if the exception is
acceptable. I suspect this is unlikely given that Apache Toree had a
problem with jeromq which led to jeromq very kindly relicensing to MPL (
https://github.com/zeromq/jeromq/issues/326).
2) Request libzmq relicense to MPL. This is something the project has
begun, but seems to be in frozen currently (unless I'm missing recent
activity).
3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
4) Switching MXNet to use something other than ps-lite? (Not sure if that's
easier than #3).
Any thoughts on #3 + #4?
Thanks,
Hen
Re: ZeroMQ dependency
Posted by Isabel Drost-Fromm <is...@isabel-drost-fromm.de>.
One comment to 2: do you have any pointers to this thread? Maybe a request from a user of the project helps revive that discussion.
Not sure if that helps, but as a bit of background, at least Pieter who designed the zeromq *community* was fine with mpl iirc: http://hintjens.com/blog:74 - can't ask him anymore though ;(
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
Re: ZeroMQ dependency
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
> Nanomsg seems not be active these days. I prefer to have use raw socket for communication.
It sounds reasonable to me.
If ps have a plan to use Intel DPDK, it's a good option to consider
seastar framework. It's distributed under ALv2.
https://github.com/scylladb/seastar
Thanks
- Tsuyoshi
On Thu, Feb 23, 2017 at 8:21 AM, Li, Mu <ml...@amazon.com> wrote:
> Nanomsg seems not be active these days. I prefer to have use raw socket for communication.
>
> One question I have is that, zeromq indicates we can link against zeromq if no change is made.
>
> I think our situation means this requirement, the only place we used zeromq is this file: https://github.com/dmlc/ps-lite/blob/master/src/zmq_van.h. Then we download zermq and compile it during make: https://github.com/dmlc/ps-lite/blob/master/make/deps.mk#L21
>
> Best
> Mu
> On Feb 22, 2017, at 1:24 AM, Henri Yandell <ba...@apache.org>> wrote:
>
> It's sounding like nanomsg is the direction to go. Any other thoughts?
>
> On Tue, Feb 21, 2017 at 2:06 PM, Minjie Wang <mi...@nyu.edu>> wrote:
>
> My colleagues and I are working on adding more support for data
> transmission. For example, send/recv operators in dataflow graph to support
> some fancier parallelism. I feel like this could be part of mxnet in the
> future (need more discussions for sure). Currently, we are using zeromq
> since we already depend on it. Does that mean we should actually consider
> using other libraries?
>
> For 3), it should not be hard since we are only transmitting arrays. We
> don't need to support rich types and complex data structures like normal
> RPC.
>
> - Minjie
>
> On Tue, Feb 21, 2017 at 11:53 AM, Will, Martin <ha...@amazon.com>>
> wrote:
>
> Re 3.) Nanomsg is licensed under BSD. [http://nanomsg.org/]. It’s
> written
> by one of the original authors of zeromq, and can be considered as an
> evolution it. The API mostly maps 1-to-1.
>
> - Martin
>
>
> On 2/20/17, 11:54 PM, "Henri Yandell" <ba...@apache.org>> wrote:
>
> How tied is MXNet to ZeroMQ?
>
> My notes are that ps-lite depends on it.
>
> Options I can see here are:
>
> 1) Discuss on general@incubator and determine if the exception is
> acceptable. I suspect this is unlikely given that Apache Toree had a
> problem with jeromq which led to jeromq very kindly relicensing to
> MPL
> (
> https://github.com/zeromq/jeromq/issues/326).
> 2) Request libzmq relicense to MPL. This is something the project has
> begun, but seems to be in frozen currently (unless I'm missing recent
> activity).
> 3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
> 4) Switching MXNet to use something other than ps-lite? (Not sure if
> that's
> easier than #3).
>
> Any thoughts on #3 + #4?
>
> Thanks,
>
> Hen
>
>
>
>
>
> --
> Minjie Wang
> *New York University | Computer Science*
> 715 Broadway, New York, NY, 10009
>
>
Re: ZeroMQ dependency
Posted by "Li, Mu" <ml...@amazon.com>.
Nanomsg seems not be active these days. I prefer to have use raw socket for communication.
One question I have is that, zeromq indicates we can link against zeromq if no change is made.
I think our situation means this requirement, the only place we used zeromq is this file: https://github.com/dmlc/ps-lite/blob/master/src/zmq_van.h. Then we download zermq and compile it during make: https://github.com/dmlc/ps-lite/blob/master/make/deps.mk#L21
Best
Mu
On Feb 22, 2017, at 1:24 AM, Henri Yandell <ba...@apache.org>> wrote:
It's sounding like nanomsg is the direction to go. Any other thoughts?
On Tue, Feb 21, 2017 at 2:06 PM, Minjie Wang <mi...@nyu.edu>> wrote:
My colleagues and I are working on adding more support for data
transmission. For example, send/recv operators in dataflow graph to support
some fancier parallelism. I feel like this could be part of mxnet in the
future (need more discussions for sure). Currently, we are using zeromq
since we already depend on it. Does that mean we should actually consider
using other libraries?
For 3), it should not be hard since we are only transmitting arrays. We
don't need to support rich types and complex data structures like normal
RPC.
- Minjie
On Tue, Feb 21, 2017 at 11:53 AM, Will, Martin <ha...@amazon.com>>
wrote:
Re 3.) Nanomsg is licensed under BSD. [http://nanomsg.org/]. It’s
written
by one of the original authors of zeromq, and can be considered as an
evolution it. The API mostly maps 1-to-1.
- Martin
On 2/20/17, 11:54 PM, "Henri Yandell" <ba...@apache.org>> wrote:
How tied is MXNet to ZeroMQ?
My notes are that ps-lite depends on it.
Options I can see here are:
1) Discuss on general@incubator and determine if the exception is
acceptable. I suspect this is unlikely given that Apache Toree had a
problem with jeromq which led to jeromq very kindly relicensing to
MPL
(
https://github.com/zeromq/jeromq/issues/326).
2) Request libzmq relicense to MPL. This is something the project has
begun, but seems to be in frozen currently (unless I'm missing recent
activity).
3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
4) Switching MXNet to use something other than ps-lite? (Not sure if
that's
easier than #3).
Any thoughts on #3 + #4?
Thanks,
Hen
--
Minjie Wang
*New York University | Computer Science*
715 Broadway, New York, NY, 10009
Re: ZeroMQ dependency
Posted by Henri Yandell <ba...@apache.org>.
It's sounding like nanomsg is the direction to go. Any other thoughts?
On Tue, Feb 21, 2017 at 2:06 PM, Minjie Wang <mi...@nyu.edu> wrote:
> My colleagues and I are working on adding more support for data
> transmission. For example, send/recv operators in dataflow graph to support
> some fancier parallelism. I feel like this could be part of mxnet in the
> future (need more discussions for sure). Currently, we are using zeromq
> since we already depend on it. Does that mean we should actually consider
> using other libraries?
>
> For 3), it should not be hard since we are only transmitting arrays. We
> don't need to support rich types and complex data structures like normal
> RPC.
>
> - Minjie
>
> On Tue, Feb 21, 2017 at 11:53 AM, Will, Martin <ha...@amazon.com>
> wrote:
>
> > Re 3.) Nanomsg is licensed under BSD. [http://nanomsg.org/]. It’s
> written
> > by one of the original authors of zeromq, and can be considered as an
> > evolution it. The API mostly maps 1-to-1.
> >
> > - Martin
> >
> >
> > On 2/20/17, 11:54 PM, "Henri Yandell" <ba...@apache.org> wrote:
> >
> > How tied is MXNet to ZeroMQ?
> >
> > My notes are that ps-lite depends on it.
> >
> > Options I can see here are:
> >
> > 1) Discuss on general@incubator and determine if the exception is
> > acceptable. I suspect this is unlikely given that Apache Toree had a
> > problem with jeromq which led to jeromq very kindly relicensing to
> MPL
> > (
> > https://github.com/zeromq/jeromq/issues/326).
> > 2) Request libzmq relicense to MPL. This is something the project has
> > begun, but seems to be in frozen currently (unless I'm missing recent
> > activity).
> > 3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
> > 4) Switching MXNet to use something other than ps-lite? (Not sure if
> > that's
> > easier than #3).
> >
> > Any thoughts on #3 + #4?
> >
> > Thanks,
> >
> > Hen
> >
> >
> >
>
>
> --
> Minjie Wang
> *New York University | Computer Science*
> 715 Broadway, New York, NY, 10009
>
Re: ZeroMQ dependency
Posted by Felix Cheung <fe...@hotmail.com>.
One way to mitigate the legal requirements would be to build an abstraction around messaging to support multiple messaging systems and make the GPL component a non-default, optional, user-downloadable dependencies.
________________________________
From: Minjie Wang <mi...@nyu.edu>
Sent: Tuesday, February 21, 2017 2:06:00 PM
To: dev@mxnet.incubator.apache.org
Subject: Re: ZeroMQ dependency
My colleagues and I are working on adding more support for data
transmission. For example, send/recv operators in dataflow graph to support
some fancier parallelism. I feel like this could be part of mxnet in the
future (need more discussions for sure). Currently, we are using zeromq
since we already depend on it. Does that mean we should actually consider
using other libraries?
For 3), it should not be hard since we are only transmitting arrays. We
don't need to support rich types and complex data structures like normal
RPC.
- Minjie
On Tue, Feb 21, 2017 at 11:53 AM, Will, Martin <ha...@amazon.com> wrote:
> Re 3.) Nanomsg is licensed under BSD. [http://nanomsg.org/]. It’s written
> by one of the original authors of zeromq, and can be considered as an
> evolution it. The API mostly maps 1-to-1.
>
> - Martin
>
>
> On 2/20/17, 11:54 PM, "Henri Yandell" <ba...@apache.org> wrote:
>
> How tied is MXNet to ZeroMQ?
>
> My notes are that ps-lite depends on it.
>
> Options I can see here are:
>
> 1) Discuss on general@incubator and determine if the exception is
> acceptable. I suspect this is unlikely given that Apache Toree had a
> problem with jeromq which led to jeromq very kindly relicensing to MPL
> (
> https://github.com/zeromq/jeromq/issues/326).
> 2) Request libzmq relicense to MPL. This is something the project has
> begun, but seems to be in frozen currently (unless I'm missing recent
> activity).
> 3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
> 4) Switching MXNet to use something other than ps-lite? (Not sure if
> that's
> easier than #3).
>
> Any thoughts on #3 + #4?
>
> Thanks,
>
> Hen
>
>
>
--
Minjie Wang
*New York University | Computer Science*
715 Broadway, New York, NY, 10009
Re: ZeroMQ dependency
Posted by Minjie Wang <mi...@nyu.edu>.
My colleagues and I are working on adding more support for data
transmission. For example, send/recv operators in dataflow graph to support
some fancier parallelism. I feel like this could be part of mxnet in the
future (need more discussions for sure). Currently, we are using zeromq
since we already depend on it. Does that mean we should actually consider
using other libraries?
For 3), it should not be hard since we are only transmitting arrays. We
don't need to support rich types and complex data structures like normal
RPC.
- Minjie
On Tue, Feb 21, 2017 at 11:53 AM, Will, Martin <ha...@amazon.com> wrote:
> Re 3.) Nanomsg is licensed under BSD. [http://nanomsg.org/]. It’s written
> by one of the original authors of zeromq, and can be considered as an
> evolution it. The API mostly maps 1-to-1.
>
> - Martin
>
>
> On 2/20/17, 11:54 PM, "Henri Yandell" <ba...@apache.org> wrote:
>
> How tied is MXNet to ZeroMQ?
>
> My notes are that ps-lite depends on it.
>
> Options I can see here are:
>
> 1) Discuss on general@incubator and determine if the exception is
> acceptable. I suspect this is unlikely given that Apache Toree had a
> problem with jeromq which led to jeromq very kindly relicensing to MPL
> (
> https://github.com/zeromq/jeromq/issues/326).
> 2) Request libzmq relicense to MPL. This is something the project has
> begun, but seems to be in frozen currently (unless I'm missing recent
> activity).
> 3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
> 4) Switching MXNet to use something other than ps-lite? (Not sure if
> that's
> easier than #3).
>
> Any thoughts on #3 + #4?
>
> Thanks,
>
> Hen
>
>
>
--
Minjie Wang
*New York University | Computer Science*
715 Broadway, New York, NY, 10009
Re: ZeroMQ dependency
Posted by "Will, Martin" <ha...@amazon.com>.
Re 3.) Nanomsg is licensed under BSD. [http://nanomsg.org/]. It’s written by one of the original authors of zeromq, and can be considered as an evolution it. The API mostly maps 1-to-1.
- Martin
On 2/20/17, 11:54 PM, "Henri Yandell" <ba...@apache.org> wrote:
How tied is MXNet to ZeroMQ?
My notes are that ps-lite depends on it.
Options I can see here are:
1) Discuss on general@incubator and determine if the exception is
acceptable. I suspect this is unlikely given that Apache Toree had a
problem with jeromq which led to jeromq very kindly relicensing to MPL (
https://github.com/zeromq/jeromq/issues/326).
2) Request libzmq relicense to MPL. This is something the project has
begun, but seems to be in frozen currently (unless I'm missing recent
activity).
3) Rewrite MXNet to not rely on zeromq. How difficult would that be?
4) Switching MXNet to use something other than ps-lite? (Not sure if that's
easier than #3).
Any thoughts on #3 + #4?
Thanks,
Hen