You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Wes McKinney <we...@gmail.com> on 2021/11/18 01:54:56 UTC

[DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

hi all,

I wanted to bring up an idea that has been raised to me several times
in the context of algorithms engineering work in Apache Arrow. The
gist is that it can be challenging to do algorithms research inside
large production systems, especially for those with a certain domain
expertise (like SIMD vectorization) but perhaps are not well-versed in
the software "structures" around the algorithms that are being
studied.

Some folks in the HPC community in recent years have been creating
smaller "proxy applications", also called "MiniApps" [1], to make the
algorithms and performance research more accessible to developers who
are less familiar with the software structures and class hierarchies
that often surround the algorithms. It also allows the research work
to evolve without having to pass through the gauntlet of the
production CI systems and merge process. Eventually, research will get
copied into the production application or form the basis of a clean
implementation in production. Some Arrow contributors have already
been doing some "miniapp" type development, though in external
repositories (here's one relating to a discussion a couple of years
ago about threading programming models and libraries where we were
talking about Intel TBB: [2]).

In short I wanted to propose creating a separate git repository under
apache/arrow-* for this purpose, to invite these kinds of
contributions to our project and to help more R&D work happen inside
the Arrow umbrella so we have clean IP lineage. I can't imagine we
would ever make releases from this repository but it could serve as a
flexible place to put stuff (even in branches that are independent
from each other) that may or may not be ready to make its home in one
of our production repositories.

Thoughts welcome!

Thanks
Wes

[1]: https://www.osti.gov/servlets/purl/1461070
[2]: https://github.com/anton-malakhov/nyctaxi

Re: [DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

Posted by Micah Kornfield <em...@gmail.com>.
In general, if this is something people have energy to pursue, it could be
useful, I'm a little bit afraid of bit-rot creeping in pretty quickly, but
it seems like it is worth an experiment.

On Tue, Dec 14, 2021 at 4:24 PM Wes McKinney <we...@gmail.com> wrote:

> I created
>
> https://github.com/apache/arrow-experiments
>
> to provide a testing ground for this idea. Might need to add a README
> indicating the protocol for adding new branches (and documenting their
> existence) and making PRs.
>
> To Micah's questions
>
> > What is the expectation around code reviews for these PRs?
>
> I think from a code review standpoint the main focus should be IP
> lineage and licensing. While we aren't going to make Apache releases
> out of this repository, if code gets prototyped there and then moved
> at a later date into the main repository, we don't want to have to go
> spelunking through experiments PRs to make sure that the IP lineage is
> clean (3rd party licenses are documented, code is not copy-pasted from
> StackOverflow, etc.).
>
> > What does minimal look like here?  Would minimal look like here?
> Programmed against the C-ABI (introduce a C only API?).
>
> This is a good question. One possibility for C++ experiments is a
> pared down version of the existing C++ project containing only some
> portions of arrow/array and pieces like MemoryPool and Buffer, along
> with certain toolchain components like googletest and gbenchmark, so
> you can write some prototype code with accompanying microbenchmarks
> and/or unit tests as part of exploring something new.
>
> Other "small" APIs for experiments could be created. For example one
> of my colleagues recently made a small R package containing a
> miniature C Arrow implementation that implements the C-ABI with
> interfaces with the R C API:
>
> https://github.com/paleolimbot/carrow/tree/master/src
>
> On Fri, Nov 26, 2021 at 1:37 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > >
> > > People would make pull requests like any other repository, but it
> > > would be a bit more free form than our other repositories. The goal is
> > > to get this kind of collaboration (code and the discussions) happening
> > > on Arrow community channels.
> >
> >
> > What is the expectation around code reviews for these PRs?   What does
> > minimal look like here?  Would minimal look like here?  Programmed
> against
> > the C-ABI (introduce a C only API?).
> >
> > On Thu, Nov 18, 2021 at 9:33 AM Benson Muite <benson_muite@emailplus.org
> >
> > wrote:
> >
> > > On 11/18/21 6:29 PM, Wes McKinney wrote:
> > > > On Thu, Nov 18, 2021 at 2:25 AM Antoine Pitrou <an...@python.org>
> > > wrote:
> > > >>
> > > >>
> > > >> Le 18/11/2021 à 02:54, Wes McKinney a écrit :
> > > >>>
> > > >>> In short I wanted to propose creating a separate git repository
> under
> > > >>> apache/arrow-* for this purpose, to invite these kinds of
> > > >>> contributions to our project and to help more R&D work happen
> inside
> > > >>> the Arrow umbrella so we have clean IP lineage. I can't imagine we
> > > >>> would ever make releases from this repository but it could serve
> as a
> > > >>> flexible place to put stuff (even in branches that are independent
> > > >>> from each other) that may or may not be ready to make its home in
> one
> > > >>> of our production repositories.
> > > >>
> > > >> What would be the rules for contributing? Is it just a place where
> > > >> people store source code?
> > > >
> > > > People would make pull requests like any other repository, but it
> > > > would be a bit more free form than our other repositories. The goal
> is
> > > > to get this kind of collaboration (code and the discussions)
> happening
> > > > on Arrow community channels.
> > > This may be helpful. Some of it might also lead to developer/interested
> > > user documentation, perhaps similar to the R-Journal
> > > https://journal.r-project.org/ but with less formality
> > > >
> > >
> > >
>

Re: [DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

Posted by Wes McKinney <we...@gmail.com>.
I created

https://github.com/apache/arrow-experiments

to provide a testing ground for this idea. Might need to add a README
indicating the protocol for adding new branches (and documenting their
existence) and making PRs.

To Micah's questions

> What is the expectation around code reviews for these PRs?

I think from a code review standpoint the main focus should be IP
lineage and licensing. While we aren't going to make Apache releases
out of this repository, if code gets prototyped there and then moved
at a later date into the main repository, we don't want to have to go
spelunking through experiments PRs to make sure that the IP lineage is
clean (3rd party licenses are documented, code is not copy-pasted from
StackOverflow, etc.).

> What does minimal look like here?  Would minimal look like here?  Programmed against the C-ABI (introduce a C only API?).

This is a good question. One possibility for C++ experiments is a
pared down version of the existing C++ project containing only some
portions of arrow/array and pieces like MemoryPool and Buffer, along
with certain toolchain components like googletest and gbenchmark, so
you can write some prototype code with accompanying microbenchmarks
and/or unit tests as part of exploring something new.

Other "small" APIs for experiments could be created. For example one
of my colleagues recently made a small R package containing a
miniature C Arrow implementation that implements the C-ABI with
interfaces with the R C API:

https://github.com/paleolimbot/carrow/tree/master/src

On Fri, Nov 26, 2021 at 1:37 PM Micah Kornfield <em...@gmail.com> wrote:
>
> >
> > People would make pull requests like any other repository, but it
> > would be a bit more free form than our other repositories. The goal is
> > to get this kind of collaboration (code and the discussions) happening
> > on Arrow community channels.
>
>
> What is the expectation around code reviews for these PRs?   What does
> minimal look like here?  Would minimal look like here?  Programmed against
> the C-ABI (introduce a C only API?).
>
> On Thu, Nov 18, 2021 at 9:33 AM Benson Muite <be...@emailplus.org>
> wrote:
>
> > On 11/18/21 6:29 PM, Wes McKinney wrote:
> > > On Thu, Nov 18, 2021 at 2:25 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > >>
> > >>
> > >> Le 18/11/2021 à 02:54, Wes McKinney a écrit :
> > >>>
> > >>> In short I wanted to propose creating a separate git repository under
> > >>> apache/arrow-* for this purpose, to invite these kinds of
> > >>> contributions to our project and to help more R&D work happen inside
> > >>> the Arrow umbrella so we have clean IP lineage. I can't imagine we
> > >>> would ever make releases from this repository but it could serve as a
> > >>> flexible place to put stuff (even in branches that are independent
> > >>> from each other) that may or may not be ready to make its home in one
> > >>> of our production repositories.
> > >>
> > >> What would be the rules for contributing? Is it just a place where
> > >> people store source code?
> > >
> > > People would make pull requests like any other repository, but it
> > > would be a bit more free form than our other repositories. The goal is
> > > to get this kind of collaboration (code and the discussions) happening
> > > on Arrow community channels.
> > This may be helpful. Some of it might also lead to developer/interested
> > user documentation, perhaps similar to the R-Journal
> > https://journal.r-project.org/ but with less formality
> > >
> >
> >

Re: [DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

Posted by Micah Kornfield <em...@gmail.com>.
>
> People would make pull requests like any other repository, but it
> would be a bit more free form than our other repositories. The goal is
> to get this kind of collaboration (code and the discussions) happening
> on Arrow community channels.


What is the expectation around code reviews for these PRs?   What does
minimal look like here?  Would minimal look like here?  Programmed against
the C-ABI (introduce a C only API?).

On Thu, Nov 18, 2021 at 9:33 AM Benson Muite <be...@emailplus.org>
wrote:

> On 11/18/21 6:29 PM, Wes McKinney wrote:
> > On Thu, Nov 18, 2021 at 2:25 AM Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >>
> >> Le 18/11/2021 à 02:54, Wes McKinney a écrit :
> >>>
> >>> In short I wanted to propose creating a separate git repository under
> >>> apache/arrow-* for this purpose, to invite these kinds of
> >>> contributions to our project and to help more R&D work happen inside
> >>> the Arrow umbrella so we have clean IP lineage. I can't imagine we
> >>> would ever make releases from this repository but it could serve as a
> >>> flexible place to put stuff (even in branches that are independent
> >>> from each other) that may or may not be ready to make its home in one
> >>> of our production repositories.
> >>
> >> What would be the rules for contributing? Is it just a place where
> >> people store source code?
> >
> > People would make pull requests like any other repository, but it
> > would be a bit more free form than our other repositories. The goal is
> > to get this kind of collaboration (code and the discussions) happening
> > on Arrow community channels.
> This may be helpful. Some of it might also lead to developer/interested
> user documentation, perhaps similar to the R-Journal
> https://journal.r-project.org/ but with less formality
> >
>
>

Re: [DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

Posted by Benson Muite <be...@emailplus.org>.
On 11/18/21 6:29 PM, Wes McKinney wrote:
> On Thu, Nov 18, 2021 at 2:25 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Le 18/11/2021 à 02:54, Wes McKinney a écrit :
>>>
>>> In short I wanted to propose creating a separate git repository under
>>> apache/arrow-* for this purpose, to invite these kinds of
>>> contributions to our project and to help more R&D work happen inside
>>> the Arrow umbrella so we have clean IP lineage. I can't imagine we
>>> would ever make releases from this repository but it could serve as a
>>> flexible place to put stuff (even in branches that are independent
>>> from each other) that may or may not be ready to make its home in one
>>> of our production repositories.
>>
>> What would be the rules for contributing? Is it just a place where
>> people store source code?
> 
> People would make pull requests like any other repository, but it
> would be a bit more free form than our other repositories. The goal is
> to get this kind of collaboration (code and the discussions) happening
> on Arrow community channels.
This may be helpful. Some of it might also lead to developer/interested 
user documentation, perhaps similar to the R-Journal 
https://journal.r-project.org/ but with less formality
> 


Re: [DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

Posted by Wes McKinney <we...@gmail.com>.
On Thu, Nov 18, 2021 at 2:25 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Le 18/11/2021 à 02:54, Wes McKinney a écrit :
> >
> > In short I wanted to propose creating a separate git repository under
> > apache/arrow-* for this purpose, to invite these kinds of
> > contributions to our project and to help more R&D work happen inside
> > the Arrow umbrella so we have clean IP lineage. I can't imagine we
> > would ever make releases from this repository but it could serve as a
> > flexible place to put stuff (even in branches that are independent
> > from each other) that may or may not be ready to make its home in one
> > of our production repositories.
>
> What would be the rules for contributing? Is it just a place where
> people store source code?

People would make pull requests like any other repository, but it
would be a bit more free form than our other repositories. The goal is
to get this kind of collaboration (code and the discussions) happening
on Arrow community channels.

> Regards
>
> Antoine.

Re: [DISCUSS] A repository for collaborative prototyping + algorithms / performance research?

Posted by Antoine Pitrou <an...@python.org>.
Le 18/11/2021 à 02:54, Wes McKinney a écrit :
> 
> In short I wanted to propose creating a separate git repository under
> apache/arrow-* for this purpose, to invite these kinds of
> contributions to our project and to help more R&D work happen inside
> the Arrow umbrella so we have clean IP lineage. I can't imagine we
> would ever make releases from this repository but it could serve as a
> flexible place to put stuff (even in branches that are independent
> from each other) that may or may not be ready to make its home in one
> of our production repositories.

What would be the rules for contributing? Is it just a place where 
people store source code?

Regards

Antoine.