You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Joris Peeters <jo...@gmail.com> on 2018/02/13 13:41:54 UTC

Arrow for MATLAB?

Hello,

Is anyone aware of plans (or concrete projects) to add MATLAB bindings for
Arrow? I'm interested in exchanging data between Java, Python, ..., and
MATLAB - and Arrow sounds like a great solution.

I couldn't find any pre-existing effort, though, so curious if that is due
to a lack of interest or because there might be underlying reasons that
would make this very hard to achieve.

Best,
-Joris.

Re: Arrow for MATLAB?

Posted by Nick Haddad <Ni...@mathworks.com>.
Thanks Wes,

I will look these over!

-Nick

 
On 2/15/18, 12:45 PM, "Wes McKinney" <we...@gmail.com> wrote:

    hi Nick,
    
    I would strongly encourage you to focus your energies on supporting
    the general Arrow streaming and file formats as described in
   http://arrow.apache.org/docs/ipc.html. It's fine to support Feather,
    so long as you restrict yourself to the API defined in
    https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/feather.h
    
    As soon as we have reasonable R bindings for the Arrow C++ libraries,
    I would like to replace the internal details of the Feather format
    with the Arrow IPC format, deprecating the metadata defined in
    https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/feather.fbs
    
    Please let us know how things go and if there's anything we can do to help.
    
    Thanks,
    Wes
    
    On Thu, Feb 15, 2018 at 9:48 AM, Nick Haddad <Ni...@mathworks.com> wrote:
    > Uwe and Joris,
    >
    > I work for MathWorks. We plan to contribute Feather read/write capability in the near future. Interesting idea about MATLAB and Arrow. We’ll take a look as well.
    >
    > Thanks
    > -Nick
    >
    >
    > On 2/14/18, 3:39 AM, "Joris Peeters" <jo...@gmail.com> wrote:
    >
    >     Yeah, pinging Mathworks might be worth it. It feels like this is something
    >     that could be of great value to them. Aside from HDF5 & mat files (both on
    >     disk), it can be really tedious to efficiently share data from anywhere
    >     with MATLAB, so it's becoming increasingly isolated.
    >     I'm going to play around for a few days with Arrow & the Matlab C++ API,
    >     getting a bit more familiar with it and maybe hacking a small prototype
    >     together. Currently it's just out-of-hours, though, so don't expect major
    >     magic. :)
    >
    >     Thanks,
    >     -J
    >
    >     On Tue, Feb 13, 2018 at 2:33 PM, Phillip Cloud <cp...@gmail.com> wrote:
    >
    >     > The MathWorks is in the process of starting to contribute. I spoke with
    >     > them a couple weeks ago about this and they were excited about it. I can
    >     > ping them to see if they are still interested.
    >     >
    >     > On Tue, Feb 13, 2018, 09:24 Uwe L. Korn <uw...@xhochy.com> wrote:
    >     >
    >     > > Hello Joris,
    >     > >
    >     > > this is only due to lack of someone doing it and probably due to lack of
    >     > > people that have the experience to do that. I had a short look at
    >     > Matlab's
    >     > > C++ API and the interfaces seem to be promising enough
    >     > > https://de.mathworks.com/help/matlab/matlab-data-array.html that once
    >     > > someone attempts it, it should not be hard to build.
    >     > >
    >     > > If you want to try to take a shot, we are happy to help if there are
    >     > > problems with the Arrow side of things.
    >     > >
    >     > > Uwe
    >     > >
    >     > > On Tue, Feb 13, 2018, at 2:41 PM, Joris Peeters wrote:
    >     > > > Hello,
    >     > > >
    >     > > > Is anyone aware of plans (or concrete projects) to add MATLAB bindings
    >     > > for
    >     > > > Arrow? I'm interested in exchanging data between Java, Python, ..., and
    >     > > > MATLAB - and Arrow sounds like a great solution.
    >     > > >
    >     > > > I couldn't find any pre-existing effort, though, so curious if that is
    >     > > due
    >     > > > to a lack of interest or because there might be underlying reasons that
    >     > > > would make this very hard to achieve.
    >     > > >
    >     > > > Best,
    >     > > > -Joris.
    >     > >
    >     >
    >
    >
    
    


Re: Arrow for MATLAB?

Posted by Wes McKinney <we...@gmail.com>.
hi Nick,

I would strongly encourage you to focus your energies on supporting
the general Arrow streaming and file formats as described in
http://arrow.apache.org/docs/ipc.html. It's fine to support Feather,
so long as you restrict yourself to the API defined in
https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/feather.h

As soon as we have reasonable R bindings for the Arrow C++ libraries,
I would like to replace the internal details of the Feather format
with the Arrow IPC format, deprecating the metadata defined in
https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/feather.fbs.

Please let us know how things go and if there's anything we can do to help.

Thanks,
Wes

On Thu, Feb 15, 2018 at 9:48 AM, Nick Haddad <Ni...@mathworks.com> wrote:
> Uwe and Joris,
>
> I work for MathWorks. We plan to contribute Feather read/write capability in the near future. Interesting idea about MATLAB and Arrow. We’ll take a look as well.
>
> Thanks
> -Nick
>
>
> On 2/14/18, 3:39 AM, "Joris Peeters" <jo...@gmail.com> wrote:
>
>     Yeah, pinging Mathworks might be worth it. It feels like this is something
>     that could be of great value to them. Aside from HDF5 & mat files (both on
>     disk), it can be really tedious to efficiently share data from anywhere
>     with MATLAB, so it's becoming increasingly isolated.
>     I'm going to play around for a few days with Arrow & the Matlab C++ API,
>     getting a bit more familiar with it and maybe hacking a small prototype
>     together. Currently it's just out-of-hours, though, so don't expect major
>     magic. :)
>
>     Thanks,
>     -J
>
>     On Tue, Feb 13, 2018 at 2:33 PM, Phillip Cloud <cp...@gmail.com> wrote:
>
>     > The MathWorks is in the process of starting to contribute. I spoke with
>     > them a couple weeks ago about this and they were excited about it. I can
>     > ping them to see if they are still interested.
>     >
>     > On Tue, Feb 13, 2018, 09:24 Uwe L. Korn <uw...@xhochy.com> wrote:
>     >
>     > > Hello Joris,
>     > >
>     > > this is only due to lack of someone doing it and probably due to lack of
>     > > people that have the experience to do that. I had a short look at
>     > Matlab's
>     > > C++ API and the interfaces seem to be promising enough
>     > > https://de.mathworks.com/help/matlab/matlab-data-array.html that once
>     > > someone attempts it, it should not be hard to build.
>     > >
>     > > If you want to try to take a shot, we are happy to help if there are
>     > > problems with the Arrow side of things.
>     > >
>     > > Uwe
>     > >
>     > > On Tue, Feb 13, 2018, at 2:41 PM, Joris Peeters wrote:
>     > > > Hello,
>     > > >
>     > > > Is anyone aware of plans (or concrete projects) to add MATLAB bindings
>     > > for
>     > > > Arrow? I'm interested in exchanging data between Java, Python, ..., and
>     > > > MATLAB - and Arrow sounds like a great solution.
>     > > >
>     > > > I couldn't find any pre-existing effort, though, so curious if that is
>     > > due
>     > > > to a lack of interest or because there might be underlying reasons that
>     > > > would make this very hard to achieve.
>     > > >
>     > > > Best,
>     > > > -Joris.
>     > >
>     >
>
>

Re: Arrow for MATLAB?

Posted by Nick Haddad <Ni...@mathworks.com>.
Uwe and Joris, 

I work for MathWorks. We plan to contribute Feather read/write capability in the near future. Interesting idea about MATLAB and Arrow. We’ll take a look as well.

Thanks
-Nick


On 2/14/18, 3:39 AM, "Joris Peeters" <jo...@gmail.com> wrote:

    Yeah, pinging Mathworks might be worth it. It feels like this is something
    that could be of great value to them. Aside from HDF5 & mat files (both on
    disk), it can be really tedious to efficiently share data from anywhere
    with MATLAB, so it's becoming increasingly isolated.
    I'm going to play around for a few days with Arrow & the Matlab C++ API,
    getting a bit more familiar with it and maybe hacking a small prototype
    together. Currently it's just out-of-hours, though, so don't expect major
    magic. :)
    
    Thanks,
    -J
    
    On Tue, Feb 13, 2018 at 2:33 PM, Phillip Cloud <cp...@gmail.com> wrote:
    
    > The MathWorks is in the process of starting to contribute. I spoke with
    > them a couple weeks ago about this and they were excited about it. I can
    > ping them to see if they are still interested.
    >
    > On Tue, Feb 13, 2018, 09:24 Uwe L. Korn <uw...@xhochy.com> wrote:
    >
    > > Hello Joris,
    > >
    > > this is only due to lack of someone doing it and probably due to lack of
    > > people that have the experience to do that. I had a short look at
    > Matlab's
    > > C++ API and the interfaces seem to be promising enough
    > > https://de.mathworks.com/help/matlab/matlab-data-array.html that once
    > > someone attempts it, it should not be hard to build.
    > >
    > > If you want to try to take a shot, we are happy to help if there are
    > > problems with the Arrow side of things.
    > >
    > > Uwe
    > >
    > > On Tue, Feb 13, 2018, at 2:41 PM, Joris Peeters wrote:
    > > > Hello,
    > > >
    > > > Is anyone aware of plans (or concrete projects) to add MATLAB bindings
    > > for
    > > > Arrow? I'm interested in exchanging data between Java, Python, ..., and
    > > > MATLAB - and Arrow sounds like a great solution.
    > > >
    > > > I couldn't find any pre-existing effort, though, so curious if that is
    > > due
    > > > to a lack of interest or because there might be underlying reasons that
    > > > would make this very hard to achieve.
    > > >
    > > > Best,
    > > > -Joris.
    > >
    >
    


Re: Arrow for MATLAB?

Posted by Joris Peeters <jo...@gmail.com>.
Yeah, pinging Mathworks might be worth it. It feels like this is something
that could be of great value to them. Aside from HDF5 & mat files (both on
disk), it can be really tedious to efficiently share data from anywhere
with MATLAB, so it's becoming increasingly isolated.
I'm going to play around for a few days with Arrow & the Matlab C++ API,
getting a bit more familiar with it and maybe hacking a small prototype
together. Currently it's just out-of-hours, though, so don't expect major
magic. :)

Thanks,
-J

On Tue, Feb 13, 2018 at 2:33 PM, Phillip Cloud <cp...@gmail.com> wrote:

> The MathWorks is in the process of starting to contribute. I spoke with
> them a couple weeks ago about this and they were excited about it. I can
> ping them to see if they are still interested.
>
> On Tue, Feb 13, 2018, 09:24 Uwe L. Korn <uw...@xhochy.com> wrote:
>
> > Hello Joris,
> >
> > this is only due to lack of someone doing it and probably due to lack of
> > people that have the experience to do that. I had a short look at
> Matlab's
> > C++ API and the interfaces seem to be promising enough
> > https://de.mathworks.com/help/matlab/matlab-data-array.html that once
> > someone attempts it, it should not be hard to build.
> >
> > If you want to try to take a shot, we are happy to help if there are
> > problems with the Arrow side of things.
> >
> > Uwe
> >
> > On Tue, Feb 13, 2018, at 2:41 PM, Joris Peeters wrote:
> > > Hello,
> > >
> > > Is anyone aware of plans (or concrete projects) to add MATLAB bindings
> > for
> > > Arrow? I'm interested in exchanging data between Java, Python, ..., and
> > > MATLAB - and Arrow sounds like a great solution.
> > >
> > > I couldn't find any pre-existing effort, though, so curious if that is
> > due
> > > to a lack of interest or because there might be underlying reasons that
> > > would make this very hard to achieve.
> > >
> > > Best,
> > > -Joris.
> >
>

Re: Arrow for MATLAB?

Posted by Phillip Cloud <cp...@gmail.com>.
The MathWorks is in the process of starting to contribute. I spoke with
them a couple weeks ago about this and they were excited about it. I can
ping them to see if they are still interested.

On Tue, Feb 13, 2018, 09:24 Uwe L. Korn <uw...@xhochy.com> wrote:

> Hello Joris,
>
> this is only due to lack of someone doing it and probably due to lack of
> people that have the experience to do that. I had a short look at Matlab's
> C++ API and the interfaces seem to be promising enough
> https://de.mathworks.com/help/matlab/matlab-data-array.html that once
> someone attempts it, it should not be hard to build.
>
> If you want to try to take a shot, we are happy to help if there are
> problems with the Arrow side of things.
>
> Uwe
>
> On Tue, Feb 13, 2018, at 2:41 PM, Joris Peeters wrote:
> > Hello,
> >
> > Is anyone aware of plans (or concrete projects) to add MATLAB bindings
> for
> > Arrow? I'm interested in exchanging data between Java, Python, ..., and
> > MATLAB - and Arrow sounds like a great solution.
> >
> > I couldn't find any pre-existing effort, though, so curious if that is
> due
> > to a lack of interest or because there might be underlying reasons that
> > would make this very hard to achieve.
> >
> > Best,
> > -Joris.
>

Re: Arrow for MATLAB?

Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Hello Joris,

this is only due to lack of someone doing it and probably due to lack of people that have the experience to do that. I had a short look at Matlab's C++ API and the interfaces seem to be promising enough https://de.mathworks.com/help/matlab/matlab-data-array.html that once someone attempts it, it should not be hard to build.

If you want to try to take a shot, we are happy to help if there are problems with the Arrow side of things.

Uwe

On Tue, Feb 13, 2018, at 2:41 PM, Joris Peeters wrote:
> Hello,
> 
> Is anyone aware of plans (or concrete projects) to add MATLAB bindings for
> Arrow? I'm interested in exchanging data between Java, Python, ..., and
> MATLAB - and Arrow sounds like a great solution.
> 
> I couldn't find any pre-existing effort, though, so curious if that is due
> to a lack of interest or because there might be underlying reasons that
> would make this very hard to achieve.
> 
> Best,
> -Joris.