You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by "Singh, Rajan" <sr...@amazon.com> on 2018/05/23 18:05:57 UTC
MXNet Protobuf dependency
Hi,
Currently, MXNet has Protobuf ( version 2.5) as one of its dependency. The dependency comes from PS-lite<https://github.com/dmlc/ps-lite/blob/a6dda54604a07d1fb21b016ed1e3f4246b08222a/make/deps.mk#L11> used for distributed training.
Recently, we have added ONNX support in MXNet(1.2.0) contrib package( import ONNX support). This module has a runtime dependency on Protobuf(version 3) , needed for ONNX.
So, if a user tries to do “import onnx”, will get a message:
“To use this module developers need to install ONNX, which requires the protobuf compiler to be installed separately. Please follow the instructions to install ONNX and its dependencies<https://github.com/onnx/onnx#installation>. MXNet currently supports ONNX v1.1.1. Once installed, you can go through the tutorials on how to use this module.”
User will end up installing protobuf version 3.5.2. Since Protobuf backward compatibility is flaky, anything dependent on version < 2.6, will probably break. In this case, distributed training might break for the user.
IMO, To resolve this dependency conflict in MXNet, would require an update to PS-lite dependency to Protobuf version 3. Is there a POA to update this dependency for PS-lite?
FYI: We are also working on adding an export module support, will export MXNet models to ONNX format, which will also have Protobuf version 3 and ONNX as its runtime dependency.
Please let me know, what should be best path moving forward.
Thanks
Rajan
RE: MXNet Protobuf dependency
Posted by "Lv, Tao A" <ta...@intel.com>.
Yes. https://github.com/dmlc/ps-lite/pull/137
-----Original Message-----
From: Hen [mailto:bayard@apache.org]
Sent: Thursday, May 24, 2018 11:47 AM
To: dev@mxnet.incubator.apache.org
Subject: Re: MXNet Protobuf dependency
Have they opened a PR with the ps-lite project?
On Wed, May 23, 2018 at 1:38 PM Singh, Rajan <sr...@amazon.com> wrote:
> Awesome !!!!
>
> Thanks Rahul for the info. Will align
>
> Thanks
> Rajan
>
> On 5/23/18, 12:04 PM, "Rahul Huilgol" <ra...@gmail.com> wrote:
>
> Hi Rajan,
>
> This PR from the Intel folks is adding support for MPI based
> distributed
> training. They also needed proto3 and have updated the current ps-lite
> proto file to work with protobuf3.5. You might want to take a look
> at that
> and align efforts with that approach.
>
> https://github.com/apache/incubator-mxnet/pull/10696
>
> The ps-lite change:
>
> https://github.com/threeleafzerg/ps-lite/compare/a6dda54604a07d1fb21b0
> 16ed1e3f4246b08222a...a470d2270d4af4badf4c94eab9559811697332e3#diff-ba
> 121c714260f51ca98d51a080880b6d
>
> Regards,
> Rahul
>
> On Wed, 23 May 2018 at 11:06 Singh, Rajan <sr...@amazon.com> wrote:
>
> > Hi,
> >
> > Currently, MXNet has Protobuf ( version 2.5) as one of its
> dependency. The
> > dependency comes from PS-lite<
> >
> https://github.com/dmlc/ps-lite/blob/a6dda54604a07d1fb21b016ed1e3f4246
> b08222a/make/deps.mk#L11
> >
> > used for distributed training.
> > Recently, we have added ONNX support in MXNet(1.2.0) contrib package(
> > import ONNX support). This module has a runtime dependency on
> > Protobuf(version 3) , needed for ONNX.
> > So, if a user tries to do “import onnx”, will get a message:
> >
> > “To use this module developers need to install ONNX, which
> requires the
> > protobuf compiler to be installed separately. Please follow the
> > instructions to install ONNX and its dependencies<
> > https://github.com/onnx/onnx#installation>. MXNet currently
> supports ONNX
> > v1.1.1. Once installed, you can go through the tutorials on how
> to use this
> > module.”
> >
> > User will end up installing protobuf version 3.5.2. Since Protobuf
> > backward compatibility is flaky, anything dependent on version <
> 2.6, will
> > probably break. In this case, distributed training might break
> for the user.
> >
> > IMO, To resolve this dependency conflict in MXNet, would require
> an update
> > to PS-lite dependency to Protobuf version 3. Is there a POA to
> update this
> > dependency for PS-lite?
> > FYI: We are also working on adding an export module support,
> will export
> > MXNet models to ONNX format, which will also have Protobuf
> version 3 and
> > ONNX as its runtime dependency.
> >
> > Please let me know, what should be best path moving forward.
> >
> > Thanks
> > Rajan
> >
> >
>
>
>
Re: MXNet Protobuf dependency
Posted by Hen <ba...@apache.org>.
Have they opened a PR with the ps-lite project?
On Wed, May 23, 2018 at 1:38 PM Singh, Rajan <sr...@amazon.com> wrote:
> Awesome !!!!
>
> Thanks Rahul for the info. Will align
>
> Thanks
> Rajan
>
> On 5/23/18, 12:04 PM, "Rahul Huilgol" <ra...@gmail.com> wrote:
>
> Hi Rajan,
>
> This PR from the Intel folks is adding support for MPI based
> distributed
> training. They also needed proto3 and have updated the current ps-lite
> proto file to work with protobuf3.5. You might want to take a look at
> that
> and align efforts with that approach.
>
> https://github.com/apache/incubator-mxnet/pull/10696
>
> The ps-lite change:
>
> https://github.com/threeleafzerg/ps-lite/compare/a6dda54604a07d1fb21b016ed1e3f4246b08222a...a470d2270d4af4badf4c94eab9559811697332e3#diff-ba121c714260f51ca98d51a080880b6d
>
> Regards,
> Rahul
>
> On Wed, 23 May 2018 at 11:06 Singh, Rajan <sr...@amazon.com> wrote:
>
> > Hi,
> >
> > Currently, MXNet has Protobuf ( version 2.5) as one of its
> dependency. The
> > dependency comes from PS-lite<
> >
> https://github.com/dmlc/ps-lite/blob/a6dda54604a07d1fb21b016ed1e3f4246b08222a/make/deps.mk#L11
> >
> > used for distributed training.
> > Recently, we have added ONNX support in MXNet(1.2.0) contrib package(
> > import ONNX support). This module has a runtime dependency on
> > Protobuf(version 3) , needed for ONNX.
> > So, if a user tries to do “import onnx”, will get a message:
> >
> > “To use this module developers need to install ONNX, which requires
> the
> > protobuf compiler to be installed separately. Please follow the
> > instructions to install ONNX and its dependencies<
> > https://github.com/onnx/onnx#installation>. MXNet currently
> supports ONNX
> > v1.1.1. Once installed, you can go through the tutorials on how to
> use this
> > module.”
> >
> > User will end up installing protobuf version 3.5.2. Since Protobuf
> > backward compatibility is flaky, anything dependent on version <
> 2.6, will
> > probably break. In this case, distributed training might break for
> the user.
> >
> > IMO, To resolve this dependency conflict in MXNet, would require an
> update
> > to PS-lite dependency to Protobuf version 3. Is there a POA to
> update this
> > dependency for PS-lite?
> > FYI: We are also working on adding an export module support, will
> export
> > MXNet models to ONNX format, which will also have Protobuf version 3
> and
> > ONNX as its runtime dependency.
> >
> > Please let me know, what should be best path moving forward.
> >
> > Thanks
> > Rajan
> >
> >
>
>
>
Re: MXNet Protobuf dependency
Posted by "Singh, Rajan" <sr...@amazon.com>.
Awesome !!!!
Thanks Rahul for the info. Will align
Thanks
Rajan
On 5/23/18, 12:04 PM, "Rahul Huilgol" <ra...@gmail.com> wrote:
Hi Rajan,
This PR from the Intel folks is adding support for MPI based distributed
training. They also needed proto3 and have updated the current ps-lite
proto file to work with protobuf3.5. You might want to take a look at that
and align efforts with that approach.
https://github.com/apache/incubator-mxnet/pull/10696
The ps-lite change:
https://github.com/threeleafzerg/ps-lite/compare/a6dda54604a07d1fb21b016ed1e3f4246b08222a...a470d2270d4af4badf4c94eab9559811697332e3#diff-ba121c714260f51ca98d51a080880b6d
Regards,
Rahul
On Wed, 23 May 2018 at 11:06 Singh, Rajan <sr...@amazon.com> wrote:
> Hi,
>
> Currently, MXNet has Protobuf ( version 2.5) as one of its dependency. The
> dependency comes from PS-lite<
> https://github.com/dmlc/ps-lite/blob/a6dda54604a07d1fb21b016ed1e3f4246b08222a/make/deps.mk#L11>
> used for distributed training.
> Recently, we have added ONNX support in MXNet(1.2.0) contrib package(
> import ONNX support). This module has a runtime dependency on
> Protobuf(version 3) , needed for ONNX.
> So, if a user tries to do “import onnx”, will get a message:
>
> “To use this module developers need to install ONNX, which requires the
> protobuf compiler to be installed separately. Please follow the
> instructions to install ONNX and its dependencies<
> https://github.com/onnx/onnx#installation>. MXNet currently supports ONNX
> v1.1.1. Once installed, you can go through the tutorials on how to use this
> module.”
>
> User will end up installing protobuf version 3.5.2. Since Protobuf
> backward compatibility is flaky, anything dependent on version < 2.6, will
> probably break. In this case, distributed training might break for the user.
>
> IMO, To resolve this dependency conflict in MXNet, would require an update
> to PS-lite dependency to Protobuf version 3. Is there a POA to update this
> dependency for PS-lite?
> FYI: We are also working on adding an export module support, will export
> MXNet models to ONNX format, which will also have Protobuf version 3 and
> ONNX as its runtime dependency.
>
> Please let me know, what should be best path moving forward.
>
> Thanks
> Rajan
>
>
Re: MXNet Protobuf dependency
Posted by Rahul Huilgol <ra...@gmail.com>.
Hi Rajan,
This PR from the Intel folks is adding support for MPI based distributed
training. They also needed proto3 and have updated the current ps-lite
proto file to work with protobuf3.5. You might want to take a look at that
and align efforts with that approach.
https://github.com/apache/incubator-mxnet/pull/10696
The ps-lite change:
https://github.com/threeleafzerg/ps-lite/compare/a6dda54604a07d1fb21b016ed1e3f4246b08222a...a470d2270d4af4badf4c94eab9559811697332e3#diff-ba121c714260f51ca98d51a080880b6d
Regards,
Rahul
On Wed, 23 May 2018 at 11:06 Singh, Rajan <sr...@amazon.com> wrote:
> Hi,
>
> Currently, MXNet has Protobuf ( version 2.5) as one of its dependency. The
> dependency comes from PS-lite<
> https://github.com/dmlc/ps-lite/blob/a6dda54604a07d1fb21b016ed1e3f4246b08222a/make/deps.mk#L11>
> used for distributed training.
> Recently, we have added ONNX support in MXNet(1.2.0) contrib package(
> import ONNX support). This module has a runtime dependency on
> Protobuf(version 3) , needed for ONNX.
> So, if a user tries to do “import onnx”, will get a message:
>
> “To use this module developers need to install ONNX, which requires the
> protobuf compiler to be installed separately. Please follow the
> instructions to install ONNX and its dependencies<
> https://github.com/onnx/onnx#installation>. MXNet currently supports ONNX
> v1.1.1. Once installed, you can go through the tutorials on how to use this
> module.”
>
> User will end up installing protobuf version 3.5.2. Since Protobuf
> backward compatibility is flaky, anything dependent on version < 2.6, will
> probably break. In this case, distributed training might break for the user.
>
> IMO, To resolve this dependency conflict in MXNet, would require an update
> to PS-lite dependency to Protobuf version 3. Is there a POA to update this
> dependency for PS-lite?
> FYI: We are also working on adding an export module support, will export
> MXNet models to ONNX format, which will also have Protobuf version 3 and
> ONNX as its runtime dependency.
>
> Please let me know, what should be best path moving forward.
>
> Thanks
> Rajan
>
>