Re: Adding CudaBuffer to pyarrow

[Pearu Peterson <pe...@quansight.com>]

Hi all,

Please review the PR for enabling GPU support in pyarrow:

https://github.com/apache/arrow/pull/2536

Best regards,
Pearu

On Wed, Aug 29, 2018 at 6:45 PM, Antoine Pitrou <an...@python.org> wrote:

>
> Hi Pearu,
>
> Your approach seems ok on the principle.  Feel free to open a PR once
> you have a basic API defined and exposed, so that we can iterate on that.
>
> Regards
>
> Antoine.
>
>
> Le 29/08/2018 à 17:11, Pearu Peterson a écrit :
> > Hi Wes,
> >
> > Thanks for the hints!
> >
> > I have created a very minimal GPU support to pyarrow, see
> >
> >
> > https://github.com/apache/arrow/compare/master...
> Quansight:pearu-cuda-pyarrow?expand=1
> >
> > This includes:
> > 1. introducing libarrow_gpu that is built into pyarrow.lib_gpu extension
> > module, currently only CudaBuffer is exposed
> > 2. updating cmake configuration files to detect libarrow_gpu.{so,a}
> > libraries
> > 3. updating setup.py with new options --with{,out}-arrow-gpu and
> > PYARROW_BUILD_ARROW_GPU env variable support.
> >
> > Let me know if something in my approach needs to be changed.
> >
> > Regarding packaging:
> > 1. FYI, there is a cudatoolkit-dev package in conda-forge:
> >   https://github.com/conda-forge/cudatoolkit-dev-feedstock
> >
> > 2. One approach is to introduce new conda packages arrow-cpp-gpu and
> > pyarrow-gpu that are build using -DARROW_GPU and --with-arrow-gpu,
> > respectively.
> >
> > Best regards,
> > Pearu
> >
> > On Wed, Aug 29, 2018 at 12:32 AM, Wes McKinney <we...@gmail.com>
> wrote:
> >
> >> hi Pearu,
> >>
> >> Sounds good to me; I had wanted to do it already but got caught up
> >> working on other things. You'll probably want to create
> >> libarrow_gpu.pxd since this is an optional library. Otherwise it
> >> should be reasonably straightforward.
> >>
> >> some relevant issues
> >>
> >> - https://issues.apache.org/jira/browse/ARROW-1424
> >> - https://issues.apache.org/jira/browse/ARROW-2512
> >>
> >> If we wanted to ship Python wheels and conda packages with CUDA
> >> support built in, what would we have to do (currently these binaries
> >> are produced in Circle CI, Travis CI, and Appveyor)?
> >>
> >> - Wes
> >> On Tue, Aug 28, 2018 at 4:34 PM Pearu Peterson
> >> <pe...@quansight.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Is there any interest in implementing CudaBuffer support in pyarrow?
> Are
> >>> there any obstacles that would prevent to do this?
> >>>
> >>> I would be willing to give it a try.
> >>>
> >>> As I understand, the entry point for this task would be to patch
> >>> pyarrow/includes/libarrow.pxd . Any hints for what to take into account
> >> are
> >>> appreciated.
> >>>
> >>> Best regards,
> >>> Pearu
> >>
> >
>