You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Chris Olivier <cj...@gmail.com> on 2018/03/09 18:20:44 UTC
Cython Document iteration
I have generated some cython documentation based upon work so far regarding
build system, profiling, development, debugging, etc.
https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvVrVGoYdiU3hNsqc
-Chris
Re: Cython Document iteration
Posted by Tianqi Chen <tq...@cs.washington.edu>.
Let me share one anecdote: Minerva, one of the predecessor of MxNet uses
cython to wrap C++ API, then we decided to switch to C API in MxNet, partly
due to some pains expressed by the MXNet members in Minerva team.
Tianqi
On Wed, Mar 14, 2018 at 8:40 AM, Tianqi Chen <tq...@cs.washington.edu>
wrote:
> My last email was send in the middle of typing so I am resending this
>
> Is the complexity of adding the Cython code generation and compilation
> step considered? Will it work in all platforms?
>
> The complexity in terms of compiler support for pybind11 is same as
> cython. If the users can build pybind, then likely cython works(because
> cython compiles to c that have same dependency as pybind11 does). What
> works for cython may not work for pybind, if there is a prebuild dll(say
> in windows via MSVC) and we want to use a different compiler (MinGW) to
> build
>
>
> What are the pitfalls and tradeoffs. Isn't it simpler to just hook up in
> the C++ API without going through an additional compiler?
>
> Note that both cython and pybind can wrap c++ API directly, whether using
> C or C++ API might worth another thread. But my general opinion is that
> wrapping a C API gives you clear thought on what you need and what is
> being exposed, while wrap C++ is kinda of the equivalent of wrapping a C
> version except that you don't know what is going on.
>
> The main reason why cython is preferred is that it allows easy mix with
> python ctypes code, which means we don't have to implement all the
> component in cython, but still can use ctypes to implement most things, and
> even support cases when cython is not available.
>
> Tianqi
>
>
> On Wed, Mar 14, 2018 at 8:38 AM, Tianqi Chen <tq...@cs.washington.edu>
> wrote:
>
>> Is the complexity of adding the Cython code generation and compilation
>> step considered? Will it work in all platforms?
>>
>> The complexity in terms of compiler support for pybind11 is same as
>> cython. If the users can build pybind, then likely cython works(because
>> cython compiles to c that have same dependency as pybind11 does). What
>> works for cython may not work for pybind, if there is a prebuild dll(say
>> in windows via MSVC) and we want to use a different compiler (MinGW) to
>> build
>>
>>
>> What are the pitfalls and tradeoffs. Isn't it simpler to just hook up in
>> the C++ API without going through an additional compiler?
>>
>> Note that both cython and pybind can wrap c++ API directly, whether using
>> C or C++ API might worth another thread. But my general opinion is that
>> wrapping a C API gives you clear thought on what you need and what is being
>> exposed, while C++ API
>>
>>
>> Tiqni
>>
>>
>> On Wed, Mar 14, 2018 at 5:05 AM, Pedro Larroy <
>> pedro.larroy.lists@gmail.com> wrote:
>>
>>> Nice document! I think it would be good to have a comparison to pybind11
>>> without going through the C API.
>>>
>>> Is the complexity of adding the Cython code generation and compilation
>>> step
>>> considered? Will it work in all platforms? what are the pitfalls and
>>> tradeofs. Isn't it simpler to just hook up in the C++ API without going
>>> through an additional compiler?
>>>
>>> Pedro.
>>>
>>> On Fri, Mar 9, 2018 at 7:44 PM, Chris Olivier <cj...@gmail.com>
>>> wrote:
>>>
>>> > Eric told me the current cython files "didn't work", so I have not
>>> reused
>>> > any of the previous cython attempt at this time.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk>
>>> > wrote:
>>> >
>>> > > Great! Also, what’s the current status of Cython support in MXNet? I
>>> > think
>>> > > we have the option to use Cython, which is added by Tianqi.
>>> > >
>>> > > Best,
>>> > > Xingjian
>>> > >
>>> > > Get Outlook for iOS<https://aka.ms/o0ukef>
>>> > > _____________________________
>>> > > From: Chris Olivier <cj...@gmail.com>
>>> > > Sent: Friday, March 9, 2018 10:21 AM
>>> > > Subject: Cython Document iteration
>>> > > To: <de...@mxnet.incubator.apache.org>
>>> > >
>>> > >
>>> > > I have generated some cython documentation based upon work so far
>>> > regarding
>>> > > build system, profiling, development, debugging, etc.
>>> > >
>>> > > https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
>>> > > rVGoYdiU3hNsqc
>>> > >
>>> > > -Chris
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Cython Document iteration
Posted by Tianqi Chen <tq...@cs.washington.edu>.
My last email was send in the middle of typing so I am resending this
Is the complexity of adding the Cython code generation and compilation
step considered?
Will it work in all platforms?
The complexity in terms of compiler support for pybind11 is same as cython.
If the users can build pybind, then likely cython works(because cython
compiles to c that have same dependency as pybind11 does). What works for
cython may not work for pybind, if there is a prebuild dll(say in windows
via MSVC) and we want to use a different compiler (MinGW) to build
What are the pitfalls and tradeoffs. Isn't it simpler to just hook up in the
C++ API without going through an additional compiler?
Note that both cython and pybind can wrap c++ API directly, whether using C
or C++ API might worth another thread. But my general opinion is that
wrapping a C API gives you clear thought on what you need and what is being
exposed, while wrap C++ is kinda of the equivalent of wrapping a C version
except that you don't know what is going on.
The main reason why cython is preferred is that it allows easy mix with
python ctypes code, which means we don't have to implement all the
component in cython, but still can use ctypes to implement most things, and
even support cases when cython is not available.
Tianqi
On Wed, Mar 14, 2018 at 8:38 AM, Tianqi Chen <tq...@cs.washington.edu>
wrote:
> Is the complexity of adding the Cython code generation and compilation
> step considered? Will it work in all platforms?
>
> The complexity in terms of compiler support for pybind11 is same as
> cython. If the users can build pybind, then likely cython works(because
> cython compiles to c that have same dependency as pybind11 does). What
> works for cython may not work for pybind, if there is a prebuild dll(say
> in windows via MSVC) and we want to use a different compiler (MinGW) to
> build
>
>
> What are the pitfalls and tradeoffs. Isn't it simpler to just hook up in
> the C++ API without going through an additional compiler?
>
> Note that both cython and pybind can wrap c++ API directly, whether using
> C or C++ API might worth another thread. But my general opinion is that
> wrapping a C API gives you clear thought on what you need and what is being
> exposed, while C++ API
>
>
> Tiqni
>
>
> On Wed, Mar 14, 2018 at 5:05 AM, Pedro Larroy <
> pedro.larroy.lists@gmail.com> wrote:
>
>> Nice document! I think it would be good to have a comparison to pybind11
>> without going through the C API.
>>
>> Is the complexity of adding the Cython code generation and compilation
>> step
>> considered? Will it work in all platforms? what are the pitfalls and
>> tradeofs. Isn't it simpler to just hook up in the C++ API without going
>> through an additional compiler?
>>
>> Pedro.
>>
>> On Fri, Mar 9, 2018 at 7:44 PM, Chris Olivier <cj...@gmail.com>
>> wrote:
>>
>> > Eric told me the current cython files "didn't work", so I have not
>> reused
>> > any of the previous cython attempt at this time.
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk>
>> > wrote:
>> >
>> > > Great! Also, what’s the current status of Cython support in MXNet? I
>> > think
>> > > we have the option to use Cython, which is added by Tianqi.
>> > >
>> > > Best,
>> > > Xingjian
>> > >
>> > > Get Outlook for iOS<https://aka.ms/o0ukef>
>> > > _____________________________
>> > > From: Chris Olivier <cj...@gmail.com>
>> > > Sent: Friday, March 9, 2018 10:21 AM
>> > > Subject: Cython Document iteration
>> > > To: <de...@mxnet.incubator.apache.org>
>> > >
>> > >
>> > > I have generated some cython documentation based upon work so far
>> > regarding
>> > > build system, profiling, development, debugging, etc.
>> > >
>> > > https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
>> > > rVGoYdiU3hNsqc
>> > >
>> > > -Chris
>> > >
>> > >
>> > >
>> >
>>
>
>
Re: Cython Document iteration
Posted by Tianqi Chen <tq...@cs.washington.edu>.
Is the complexity of adding the Cython code generation and compilation
step considered?
Will it work in all platforms?
The complexity in terms of compiler support for pybind11 is same as cython.
If the users can build pybind, then likely cython works(because cython
compiles to c that have same dependency as pybind11 does). What works for
cython may not work for pybind, if there is a prebuild dll(say in windows
via MSVC) and we want to use a different compiler (MinGW) to build
What are the pitfalls and tradeoffs. Isn't it simpler to just hook up in the
C++ API without going through an additional compiler?
Note that both cython and pybind can wrap c++ API directly, whether using C
or C++ API might worth another thread. But my general opinion is that
wrapping a C API gives you clear thought on what you need and what is being
exposed, while C++ API
Tiqni
On Wed, Mar 14, 2018 at 5:05 AM, Pedro Larroy <pe...@gmail.com>
wrote:
> Nice document! I think it would be good to have a comparison to pybind11
> without going through the C API.
>
> Is the complexity of adding the Cython code generation and compilation step
> considered? Will it work in all platforms? what are the pitfalls and
> tradeofs. Isn't it simpler to just hook up in the C++ API without going
> through an additional compiler?
>
> Pedro.
>
> On Fri, Mar 9, 2018 at 7:44 PM, Chris Olivier <cj...@gmail.com>
> wrote:
>
> > Eric told me the current cython files "didn't work", so I have not reused
> > any of the previous cython attempt at this time.
> >
> >
> >
> >
> >
> > On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk>
> > wrote:
> >
> > > Great! Also, what’s the current status of Cython support in MXNet? I
> > think
> > > we have the option to use Cython, which is added by Tianqi.
> > >
> > > Best,
> > > Xingjian
> > >
> > > Get Outlook for iOS<https://aka.ms/o0ukef>
> > > _____________________________
> > > From: Chris Olivier <cj...@gmail.com>
> > > Sent: Friday, March 9, 2018 10:21 AM
> > > Subject: Cython Document iteration
> > > To: <de...@mxnet.incubator.apache.org>
> > >
> > >
> > > I have generated some cython documentation based upon work so far
> > regarding
> > > build system, profiling, development, debugging, etc.
> > >
> > > https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> > > rVGoYdiU3hNsqc
> > >
> > > -Chris
> > >
> > >
> > >
> >
>
Re: Cython Document iteration
Posted by Pedro Larroy <pe...@gmail.com>.
Nice document! I think it would be good to have a comparison to pybind11
without going through the C API.
Is the complexity of adding the Cython code generation and compilation step
considered? Will it work in all platforms? what are the pitfalls and
tradeofs. Isn't it simpler to just hook up in the C++ API without going
through an additional compiler?
Pedro.
On Fri, Mar 9, 2018 at 7:44 PM, Chris Olivier <cj...@gmail.com> wrote:
> Eric told me the current cython files "didn't work", so I have not reused
> any of the previous cython attempt at this time.
>
>
>
>
>
> On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk>
> wrote:
>
> > Great! Also, what’s the current status of Cython support in MXNet? I
> think
> > we have the option to use Cython, which is added by Tianqi.
> >
> > Best,
> > Xingjian
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> > _____________________________
> > From: Chris Olivier <cj...@gmail.com>
> > Sent: Friday, March 9, 2018 10:21 AM
> > Subject: Cython Document iteration
> > To: <de...@mxnet.incubator.apache.org>
> >
> >
> > I have generated some cython documentation based upon work so far
> regarding
> > build system, profiling, development, debugging, etc.
> >
> > https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> > rVGoYdiU3hNsqc
> >
> > -Chris
> >
> >
> >
>
Re: Cython Document iteration
Posted by Chris Olivier <cj...@gmail.com>.
Eric told me the current cython files "didn't work", so I have not reused
any of the previous cython attempt at this time.
On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk> wrote:
> Great! Also, what’s the current status of Cython support in MXNet? I think
> we have the option to use Cython, which is added by Tianqi.
>
> Best,
> Xingjian
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> _____________________________
> From: Chris Olivier <cj...@gmail.com>
> Sent: Friday, March 9, 2018 10:21 AM
> Subject: Cython Document iteration
> To: <de...@mxnet.incubator.apache.org>
>
>
> I have generated some cython documentation based upon work so far regarding
> build system, profiling, development, debugging, etc.
>
> https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> rVGoYdiU3hNsqc
>
> -Chris
>
>
>
Re: Cython Document iteration
Posted by Chris Olivier <cj...@gmail.com>.
I didn't really use anything from the old cython stuff yet. Mostly setting
up build and development system and doing performance testing.
On Fri, Mar 9, 2018 at 10:32 AM, Tianqi Chen <tq...@cs.washington.edu>
wrote:
> The cython part was not actively maintained and currently disabled, but the
> code structure can likely be reused
>
> Tianqi
>
> On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk>
> wrote:
>
> > Great! Also, what’s the current status of Cython support in MXNet? I
> think
> > we have the option to use Cython, which is added by Tianqi.
> >
> > Best,
> > Xingjian
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> > _____________________________
> > From: Chris Olivier <cj...@gmail.com>
> > Sent: Friday, March 9, 2018 10:21 AM
> > Subject: Cython Document iteration
> > To: <de...@mxnet.incubator.apache.org>
> >
> >
> > I have generated some cython documentation based upon work so far
> regarding
> > build system, profiling, development, debugging, etc.
> >
> > https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> > rVGoYdiU3hNsqc
> >
> > -Chris
> >
> >
> >
>
Re: Cython Document iteration
Posted by Tianqi Chen <tq...@cs.washington.edu>.
The cython part was not actively maintained and currently disabled, but the
code structure can likely be reused
Tianqi
On Fri, Mar 9, 2018 at 10:31 AM, Xingjian SHI <xs...@connect.ust.hk> wrote:
> Great! Also, what’s the current status of Cython support in MXNet? I think
> we have the option to use Cython, which is added by Tianqi.
>
> Best,
> Xingjian
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> _____________________________
> From: Chris Olivier <cj...@gmail.com>
> Sent: Friday, March 9, 2018 10:21 AM
> Subject: Cython Document iteration
> To: <de...@mxnet.incubator.apache.org>
>
>
> I have generated some cython documentation based upon work so far regarding
> build system, profiling, development, debugging, etc.
>
> https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvV
> rVGoYdiU3hNsqc
>
> -Chris
>
>
>
Re: Cython Document iteration
Posted by Xingjian SHI <xs...@connect.ust.hk>.
Great! Also, what’s the current status of Cython support in MXNet? I think we have the option to use Cython, which is added by Tianqi.
Best,
Xingjian
Get Outlook for iOS<https://aka.ms/o0ukef>
_____________________________
From: Chris Olivier <cj...@gmail.com>
Sent: Friday, March 9, 2018 10:21 AM
Subject: Cython Document iteration
To: <de...@mxnet.incubator.apache.org>
I have generated some cython documentation based upon work so far regarding
build system, profiling, development, debugging, etc.
https://docs.google.com/document/d/1pvW5nYbf3pbiak95xZdqgLbrjQdvVrVGoYdiU3hNsqc
-Chris