You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Kevin Gurney <kg...@mathworks.com> on 2021/01/21 21:47:20 UTC

[MATLAB] Developing a MATLAB Interface for Apache Arrow

Hello All,

MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:

  *   arrow.Array
  *   arrow.TableReader
  *   arrow.type.Float64
  *   ...

In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.

Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.

It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.

We are excited to work together with the rest of the Arrow community to make this a reality.

Best Regards,

Kevin Gurney

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Kevin Gurney <kg...@mathworks.com>.
Hi All,

Just a friendly reminder that today is the soft feedback deadline mentioned in my previous email for providing feedback on the MATLAB interface design doc ( https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg ). Please feel free to comment on the design doc with any questions or concerns you have.

As feedback settles, we'll start shifting our focus towards implementation. However, the design doc will continue to remain open to community input. We'll work on migrating it to Markdown so that it can live in GitHub for long term archival purposes and easier collaboration.

If you have any questions, don't hesitate to let me know.

Thank you!

Best Regards,

Kevin Gurney
________________________________
From: Kevin Gurney <kg...@mathworks.com>
Sent: Tuesday, February 2, 2021 5:05 PM
To: dev <de...@arrow.apache.org>
Cc: Antoine Pitrou <an...@python.org>; Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>; Wes McKinney <we...@gmail.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Hi All,

Thanks again for all of the feedback we have received so far on the design doc - it's been really helpful.

Fiona (Cc'd) and I took an initial pass at addressing the comments from Wes and Antoine. @Wes and @Antoine - whenever you get a chance, we would appreciate it if you could take a look over our responses to your feedback. Don't hesitate to let us know if you have any additional questions or concerns.

Some notable changes we have made based on the feedback received so far:

    1. Added a section about the ArrowC Data Interface and In-Process Arrow Memory Sharing.
    2. Modified the Out-of-Process Arrow Memory Sharing example to use the Arrow IPC File Format rather than Feather V2.
    3. Added a note that Plasma is informally deprecated, and it may not make sense to invest effort in supporting it right now.

We continue to welcome anyone else in the community to add your thoughts and comments to the design doc.

To keep things moving along, we would like to set a "soft" deadline for feedback on this high level design doc for next **Tuesday, February 9th, 2021**. After this "soft" deadline, everyone is still free to raise concerns as they come up, but we'll transition towards focusing on initial implementation (i.e. we'll assume it is reasonable to start prototyping and opening pull requests as appropriate).

Note: We realize that Word Online isn't turning out to be an ideal solution for collaborating on design docs. The lack of support for including names with comments is proving to be cumbersome (our apologies, again). In addition, for long term archival purposes, it would be best to move the document to a better location. Fiona and I believe that using Markdown to write future design docs and user guides would be preferrable, as we can get all the benefits of normal GitHub version control, as well as have the docs live right alongside the MATLAB code. Commenting and general collaboration should be easier this way too. Our plan is to migrate this initial design doc to Markdown after feedback from the community has settled.

Thanks again to everyone for your help in getting this project off the ground!

Best Regards,

Kevin Gurney
________________________________
From: Kevin Gurney <kg...@mathworks.com>
Sent: Tuesday, January 26, 2021 11:47 AM
To: dev <de...@arrow.apache.org>
Cc: Antoine Pitrou <an...@python.org>; Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>; Wes McKinney <we...@gmail.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Hi Wes,

Thanks very much for taking the time to share your feedback!

Looking forward to incorporating more feedback from the community. Excited to work together to move this project forward!

Best Regards,

Kevin Gurney
________________________________
From: Wes McKinney <we...@gmail.com>
Sent: Monday, January 25, 2021 4:48 PM
To: dev <de...@arrow.apache.org>
Cc: Antoine Pitrou <an...@python.org>; Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

hi Kevin -- I read through the document. It seems plenty reasonable to
me. Look forward to seeing the buildout.

Thanks
Wes

On Mon, Jan 25, 2021 at 3:10 PM Kevin Gurney <kg...@mathworks.com> wrote:
>
> Hi Antoine,
>
> Thanks very much for taking a first pass over the document! I'll start working through the feedback you've provided soon.
>
> It's useful to have a variety of perspectives here, including from Arrow C++ developers, like yourself. Also, I agree - having more MATLAB users provide feedback on the document would be great to help ensure we are covering all necessary requirements to make the interface as useful as possible.
>
> If anyone has any other ideas or suggestions, please don't hesitate to share your feedback on the document.
>
> Best Regards,
>
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Monday, January 25, 2021 12:53 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
>
> Hi Kevin,
>
> I've added some comments to the document. Bear in mind that I'm not a
> MATLAB user, so this is some outside view from a Arrow C++ developer.
>
> It would be nice if MATLAB users could comment on the document,
> especially the proposed APIs.
>
> Regards
>
> Antoine.
>
>
> Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> > Hi Antoine,
> >
> > Thanks for your input!
> >
> > As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.
> >
> > Based on your suggestion, I've included a link to a Word Online version of the design document below:
> >
> > https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg><https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg>>
> >
> > As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.
> >
> > Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.
> >
> > Best Regards,
> >
> > Kevin Gurney
> > ________________________________
> > From: Antoine Pitrou <an...@python.org>
> > Sent: Friday, January 22, 2021 11:28 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> > Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> > Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >
> >
> > Hello Kevin,
> >
> > You could use a Google Docs or similar to share the design document and
> > allow people to comment. Inside a Google Doc, you can use "File ->
> > Share" to create a sharable URL with specific permissions (such as
> > commenting but not editing).
> >
> > I was about to mention the matlab/ directory in the Arrow repository but
> > I see you're the main author, so you already know about it :-)
> >
> > Best regards
> >
> > Antoine.
> >
> >
> > Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> >> It seems like the mailing list stripped out the design doc I attached for some reason.
> >>
> >> Here is a link to the same document hosted online instead:
> >>
> >> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>>>
> >>
> >> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
> >>
> >> Thanks,
> >>
> >> Kevin
> >> ________________________________
> >> From: Kevin Gurney <kg...@mathworks.com>
> >> Sent: Thursday, January 21, 2021 4:47 PM
> >> To: dev@arrow.apache.org <de...@arrow.apache.org>
> >> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> >> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >>
> >> Hello All,
> >>
> >> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
> >>
> >> * arrow.Array
> >> * arrow.TableReader
> >> * arrow.type.Float64
> >> * ...
> >>
> >> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
> >>
> >> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
> >>
> >> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>>>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
> >>
> >> We are excited to work together with the rest of the Arrow community to make this a reality.
> >>
> >> Best Regards,
> >>
> >> Kevin Gurney
> >>
> >

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Kevin Gurney <kg...@mathworks.com>.
Hi All,

Thanks again for all of the feedback we have received so far on the design doc - it's been really helpful.

Fiona (Cc'd) and I took an initial pass at addressing the comments from Wes and Antoine. @Wes and @Antoine - whenever you get a chance, we would appreciate it if you could take a look over our responses to your feedback. Don't hesitate to let us know if you have any additional questions or concerns.

Some notable changes we have made based on the feedback received so far:

    1. Added a section about the ArrowC Data Interface and In-Process Arrow Memory Sharing.
    2. Modified the Out-of-Process Arrow Memory Sharing example to use the Arrow IPC File Format rather than Feather V2.
    3. Added a note that Plasma is informally deprecated, and it may not make sense to invest effort in supporting it right now.

We continue to welcome anyone else in the community to add your thoughts and comments to the design doc.

To keep things moving along, we would like to set a "soft" deadline for feedback on this high level design doc for next **Tuesday, February 9th, 2021**. After this "soft" deadline, everyone is still free to raise concerns as they come up, but we'll transition towards focusing on initial implementation (i.e. we'll assume it is reasonable to start prototyping and opening pull requests as appropriate).

Note: We realize that Word Online isn't turning out to be an ideal solution for collaborating on design docs. The lack of support for including names with comments is proving to be cumbersome (our apologies, again). In addition, for long term archival purposes, it would be best to move the document to a better location. Fiona and I believe that using Markdown to write future design docs and user guides would be preferrable, as we can get all the benefits of normal GitHub version control, as well as have the docs live right alongside the MATLAB code. Commenting and general collaboration should be easier this way too. Our plan is to migrate this initial design doc to Markdown after feedback from the community has settled.

Thanks again to everyone for your help in getting this project off the ground!

Best Regards,

Kevin Gurney
________________________________
From: Kevin Gurney <kg...@mathworks.com>
Sent: Tuesday, January 26, 2021 11:47 AM
To: dev <de...@arrow.apache.org>
Cc: Antoine Pitrou <an...@python.org>; Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>; Wes McKinney <we...@gmail.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Hi Wes,

Thanks very much for taking the time to share your feedback!

Looking forward to incorporating more feedback from the community. Excited to work together to move this project forward!

Best Regards,

Kevin Gurney
________________________________
From: Wes McKinney <we...@gmail.com>
Sent: Monday, January 25, 2021 4:48 PM
To: dev <de...@arrow.apache.org>
Cc: Antoine Pitrou <an...@python.org>; Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

hi Kevin -- I read through the document. It seems plenty reasonable to
me. Look forward to seeing the buildout.

Thanks
Wes

On Mon, Jan 25, 2021 at 3:10 PM Kevin Gurney <kg...@mathworks.com> wrote:
>
> Hi Antoine,
>
> Thanks very much for taking a first pass over the document! I'll start working through the feedback you've provided soon.
>
> It's useful to have a variety of perspectives here, including from Arrow C++ developers, like yourself. Also, I agree - having more MATLAB users provide feedback on the document would be great to help ensure we are covering all necessary requirements to make the interface as useful as possible.
>
> If anyone has any other ideas or suggestions, please don't hesitate to share your feedback on the document.
>
> Best Regards,
>
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Monday, January 25, 2021 12:53 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
>
> Hi Kevin,
>
> I've added some comments to the document. Bear in mind that I'm not a
> MATLAB user, so this is some outside view from a Arrow C++ developer.
>
> It would be nice if MATLAB users could comment on the document,
> especially the proposed APIs.
>
> Regards
>
> Antoine.
>
>
> Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> > Hi Antoine,
> >
> > Thanks for your input!
> >
> > As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.
> >
> > Based on your suggestion, I've included a link to a Word Online version of the design document below:
> >
> > https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg><https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg>>
> >
> > As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.
> >
> > Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.
> >
> > Best Regards,
> >
> > Kevin Gurney
> > ________________________________
> > From: Antoine Pitrou <an...@python.org>
> > Sent: Friday, January 22, 2021 11:28 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> > Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> > Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >
> >
> > Hello Kevin,
> >
> > You could use a Google Docs or similar to share the design document and
> > allow people to comment. Inside a Google Doc, you can use "File ->
> > Share" to create a sharable URL with specific permissions (such as
> > commenting but not editing).
> >
> > I was about to mention the matlab/ directory in the Arrow repository but
> > I see you're the main author, so you already know about it :-)
> >
> > Best regards
> >
> > Antoine.
> >
> >
> > Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> >> It seems like the mailing list stripped out the design doc I attached for some reason.
> >>
> >> Here is a link to the same document hosted online instead:
> >>
> >> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>>>
> >>
> >> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
> >>
> >> Thanks,
> >>
> >> Kevin
> >> ________________________________
> >> From: Kevin Gurney <kg...@mathworks.com>
> >> Sent: Thursday, January 21, 2021 4:47 PM
> >> To: dev@arrow.apache.org <de...@arrow.apache.org>
> >> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> >> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >>
> >> Hello All,
> >>
> >> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
> >>
> >> * arrow.Array
> >> * arrow.TableReader
> >> * arrow.type.Float64
> >> * ...
> >>
> >> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
> >>
> >> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
> >>
> >> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>>>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
> >>
> >> We are excited to work together with the rest of the Arrow community to make this a reality.
> >>
> >> Best Regards,
> >>
> >> Kevin Gurney
> >>
> >

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Kevin Gurney <kg...@mathworks.com>.
Hi Wes,

Thanks very much for taking the time to share your feedback!

Looking forward to incorporating more feedback from the community. Excited to work together to move this project forward!

Best Regards,

Kevin Gurney
________________________________
From: Wes McKinney <we...@gmail.com>
Sent: Monday, January 25, 2021 4:48 PM
To: dev <de...@arrow.apache.org>
Cc: Antoine Pitrou <an...@python.org>; Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

hi Kevin -- I read through the document. It seems plenty reasonable to
me. Look forward to seeing the buildout.

Thanks
Wes

On Mon, Jan 25, 2021 at 3:10 PM Kevin Gurney <kg...@mathworks.com> wrote:
>
> Hi Antoine,
>
> Thanks very much for taking a first pass over the document! I'll start working through the feedback you've provided soon.
>
> It's useful to have a variety of perspectives here, including from Arrow C++ developers, like yourself. Also, I agree - having more MATLAB users provide feedback on the document would be great to help ensure we are covering all necessary requirements to make the interface as useful as possible.
>
> If anyone has any other ideas or suggestions, please don't hesitate to share your feedback on the document.
>
> Best Regards,
>
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Monday, January 25, 2021 12:53 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
>
> Hi Kevin,
>
> I've added some comments to the document. Bear in mind that I'm not a
> MATLAB user, so this is some outside view from a Arrow C++ developer.
>
> It would be nice if MATLAB users could comment on the document,
> especially the proposed APIs.
>
> Regards
>
> Antoine.
>
>
> Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> > Hi Antoine,
> >
> > Thanks for your input!
> >
> > As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.
> >
> > Based on your suggestion, I've included a link to a Word Online version of the design document below:
> >
> > https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg><https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg>>
> >
> > As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.
> >
> > Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.
> >
> > Best Regards,
> >
> > Kevin Gurney
> > ________________________________
> > From: Antoine Pitrou <an...@python.org>
> > Sent: Friday, January 22, 2021 11:28 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> > Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> > Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >
> >
> > Hello Kevin,
> >
> > You could use a Google Docs or similar to share the design document and
> > allow people to comment. Inside a Google Doc, you can use "File ->
> > Share" to create a sharable URL with specific permissions (such as
> > commenting but not editing).
> >
> > I was about to mention the matlab/ directory in the Arrow repository but
> > I see you're the main author, so you already know about it :-)
> >
> > Best regards
> >
> > Antoine.
> >
> >
> > Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> >> It seems like the mailing list stripped out the design doc I attached for some reason.
> >>
> >> Here is a link to the same document hosted online instead:
> >>
> >> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>>>
> >>
> >> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
> >>
> >> Thanks,
> >>
> >> Kevin
> >> ________________________________
> >> From: Kevin Gurney <kg...@mathworks.com>
> >> Sent: Thursday, January 21, 2021 4:47 PM
> >> To: dev@arrow.apache.org <de...@arrow.apache.org>
> >> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> >> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >>
> >> Hello All,
> >>
> >> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
> >>
> >> * arrow.Array
> >> * arrow.TableReader
> >> * arrow.type.Float64
> >> * ...
> >>
> >> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
> >>
> >> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
> >>
> >> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>>>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
> >>
> >> We are excited to work together with the rest of the Arrow community to make this a reality.
> >>
> >> Best Regards,
> >>
> >> Kevin Gurney
> >>
> >

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Wes McKinney <we...@gmail.com>.
hi Kevin -- I read through the document. It seems plenty reasonable to
me. Look forward to seeing the buildout.

Thanks
Wes

On Mon, Jan 25, 2021 at 3:10 PM Kevin Gurney <kg...@mathworks.com> wrote:
>
> Hi Antoine,
>
> Thanks very much for taking a first pass over the document! I'll start working through the feedback you've provided soon.
>
> It's useful to have a variety of perspectives here, including from Arrow C++ developers, like yourself. Also, I agree - having more MATLAB users provide feedback on the document would be great to help ensure we are covering all necessary requirements to make the interface as useful as possible.
>
> If anyone has any other ideas or suggestions, please don't hesitate to share your feedback on the document.
>
> Best Regards,
>
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Monday, January 25, 2021 12:53 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
>
> Hi Kevin,
>
> I've added some comments to the document. Bear in mind that I'm not a
> MATLAB user, so this is some outside view from a Arrow C++ developer.
>
> It would be nice if MATLAB users could comment on the document,
> especially the proposed APIs.
>
> Regards
>
> Antoine.
>
>
> Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> > Hi Antoine,
> >
> > Thanks for your input!
> >
> > As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.
> >
> > Based on your suggestion, I've included a link to a Word Online version of the design document below:
> >
> > https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg>
> >
> > As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.
> >
> > Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.
> >
> > Best Regards,
> >
> > Kevin Gurney
> > ________________________________
> > From: Antoine Pitrou <an...@python.org>
> > Sent: Friday, January 22, 2021 11:28 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> > Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> > Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >
> >
> > Hello Kevin,
> >
> > You could use a Google Docs or similar to share the design document and
> > allow people to comment. Inside a Google Doc, you can use "File ->
> > Share" to create a sharable URL with specific permissions (such as
> > commenting but not editing).
> >
> > I was about to mention the matlab/ directory in the Arrow repository but
> > I see you're the main author, so you already know about it :-)
> >
> > Best regards
> >
> > Antoine.
> >
> >
> > Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> >> It seems like the mailing list stripped out the design doc I attached for some reason.
> >>
> >> Here is a link to the same document hosted online instead:
> >>
> >> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>>
> >>
> >> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
> >>
> >> Thanks,
> >>
> >> Kevin
> >> ________________________________
> >> From: Kevin Gurney <kg...@mathworks.com>
> >> Sent: Thursday, January 21, 2021 4:47 PM
> >> To: dev@arrow.apache.org <de...@arrow.apache.org>
> >> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> >> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >>
> >> Hello All,
> >>
> >> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
> >>
> >> * arrow.Array
> >> * arrow.TableReader
> >> * arrow.type.Float64
> >> * ...
> >>
> >> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
> >>
> >> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
> >>
> >> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
> >>
> >> We are excited to work together with the rest of the Arrow community to make this a reality.
> >>
> >> Best Regards,
> >>
> >> Kevin Gurney
> >>
> >

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Kevin Gurney <kg...@mathworks.com>.
Hi Antoine,

Thanks very much for taking a first pass over the document! I'll start working through the feedback you've provided soon.

It's useful to have a variety of perspectives here, including from Arrow C++ developers, like yourself. Also, I agree - having more MATLAB users provide feedback on the document would be great to help ensure we are covering all necessary requirements to make the interface as useful as possible.

If anyone has any other ideas or suggestions, please don't hesitate to share your feedback on the document.

Best Regards,

Kevin Gurney
________________________________
From: Antoine Pitrou <an...@python.org>
Sent: Monday, January 25, 2021 12:53 PM
To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow


Hi Kevin,

I've added some comments to the document. Bear in mind that I'm not a
MATLAB user, so this is some outside view from a Arrow C++ developer.

It would be nice if MATLAB users could comment on the document,
especially the proposed APIs.

Regards

Antoine.


Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> Hi Antoine,
>
> Thanks for your input!
>
> As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.
>
> Based on your suggestion, I've included a link to a Word Online version of the design document below:
>
> https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg>
>
> As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.
>
> Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.
>
> Best Regards,
>
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Friday, January 22, 2021 11:28 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
>
> Hello Kevin,
>
> You could use a Google Docs or similar to share the design document and
> allow people to comment. Inside a Google Doc, you can use "File ->
> Share" to create a sharable URL with specific permissions (such as
> commenting but not editing).
>
> I was about to mention the matlab/ directory in the Arrow repository but
> I see you're the main author, so you already know about it :-)
>
> Best regards
>
> Antoine.
>
>
> Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
>> It seems like the mailing list stripped out the design doc I attached for some reason.
>>
>> Here is a link to the same document hosted online instead:
>>
>> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>>
>>
>> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
>>
>> Thanks,
>>
>> Kevin
>> ________________________________
>> From: Kevin Gurney <kg...@mathworks.com>
>> Sent: Thursday, January 21, 2021 4:47 PM
>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
>> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>>
>> Hello All,
>>
>> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
>>
>> * arrow.Array
>> * arrow.TableReader
>> * arrow.type.Float64
>> * ...
>>
>> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
>>
>> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
>>
>> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
>>
>> We are excited to work together with the rest of the Arrow community to make this a reality.
>>
>> Best Regards,
>>
>> Kevin Gurney
>>
>

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Antoine Pitrou <an...@python.org>.
Hi Kevin,

I've added some comments to the document.  Bear in mind that I'm not a
MATLAB user, so this is some outside view from a Arrow C++ developer.

It would be nice if MATLAB users could comment on the document,
especially the proposed APIs.

Regards

Antoine.


Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> Hi Antoine,
> 
> Thanks for your input!
> 
> As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.
> 
> Based on your suggestion, I've included a link to a Word Online version of the design document below:
> 
> https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg
> 
> As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.
> 
> Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.
> 
> Best Regards,
> 
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <an...@python.org>
> Sent: Friday, January 22, 2021 11:28 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> 
> 
> Hello Kevin,
> 
> You could use a Google Docs or similar to share the design document and
> allow people to comment. Inside a Google Doc, you can use "File ->
> Share" to create a sharable URL with specific permissions (such as
> commenting but not editing).
> 
> I was about to mention the matlab/ directory in the Arrow repository but
> I see you're the main author, so you already know about it :-)
> 
> Best regards
> 
> Antoine.
> 
> 
> Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
>> It seems like the mailing list stripped out the design doc I attached for some reason.
>>
>> Here is a link to the same document hosted online instead:
>>
>> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>
>>
>> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
>>
>> Thanks,
>>
>> Kevin
>> ________________________________
>> From: Kevin Gurney <kg...@mathworks.com>
>> Sent: Thursday, January 21, 2021 4:47 PM
>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
>> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>>
>> Hello All,
>>
>> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
>>
>> * arrow.Array
>> * arrow.TableReader
>> * arrow.type.Float64
>> * ...
>>
>> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
>>
>> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
>>
>> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
>>
>> We are excited to work together with the rest of the Arrow community to make this a reality.
>>
>> Best Regards,
>>
>> Kevin Gurney
>>
>

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Kevin Gurney <kg...@mathworks.com>.
Hi Antoine,

Thanks for your input!

As you pointed out, I am in fact familiar with the matlab/ directory! :-) Several MathWorkers, including myself, helped contribute to this code a while back. We are hoping to use it as a starting point as we build out a more fully fledged MATLAB interface to Arrow memory.

Based on your suggestion, I've included a link to a Word Online version of the design document below:

https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg

As far as I can tell, this link should allow commenting by anyone. Unfortunately, I'm not sure if the names of reviewers will be included when they comment. If this turns out to be the case, it would be great if reviewers could prefix their comments with something like [FirstName LastName] so we can track feedback appropriately.

Don't hesitate to let me know if you have any issues accessing or commenting on the document. My apologies for the inconvenience in getting this properly shared.

Best Regards,

Kevin Gurney
________________________________
From: Antoine Pitrou <an...@python.org>
Sent: Friday, January 22, 2021 11:28 AM
To: dev@arrow.apache.org <de...@arrow.apache.org>; Kevin Gurney <kg...@mathworks.com>
Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow


Hello Kevin,

You could use a Google Docs or similar to share the design document and
allow people to comment. Inside a Google Doc, you can use "File ->
Share" to create a sharable URL with specific permissions (such as
commenting but not editing).

I was about to mention the matlab/ directory in the Arrow repository but
I see you're the main author, so you already know about it :-)

Best regards

Antoine.


Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> It seems like the mailing list stripped out the design doc I attached for some reason.
>
> Here is a link to the same document hosted online instead:
>
> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>
>
> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
>
> Thanks,
>
> Kevin
> ________________________________
> From: Kevin Gurney <kg...@mathworks.com>
> Sent: Thursday, January 21, 2021 4:47 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
> Hello All,
>
> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
>
> * arrow.Array
> * arrow.TableReader
> * arrow.type.Float64
> * ...
>
> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
>
> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
>
> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
>
> We are excited to work together with the rest of the Arrow community to make this a reality.
>
> Best Regards,
>
> Kevin Gurney
>

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Antoine Pitrou <an...@python.org>.
Hello Kevin,

You could use a Google Docs or similar to share the design document and
allow people to comment. Inside a Google Doc, you can use "File ->
Share" to create a sharable URL with specific permissions (such as
commenting but not editing).

I was about to mention the matlab/ directory in the Arrow repository but
I see you're the main author, so you already know about it :-)

Best regards

Antoine.


Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> It seems like the mailing list stripped out the design doc I attached for some reason.
> 
> Here is a link to the same document hosted online instead:
> 
> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4
> 
> Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.
> 
> Thanks,
> 
> Kevin
> ________________________________
> From: Kevin Gurney <kg...@mathworks.com>
> Sent: Thursday, January 21, 2021 4:47 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> 
> Hello All,
> 
> MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:
> 
>   *   arrow.Array
>   *   arrow.TableReader
>   *   arrow.type.Float64
>   *   ...
> 
> In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
> 
> Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.
> 
> It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.
> 
> We are excited to work together with the rest of the Arrow community to make this a reality.
> 
> Best Regards,
> 
> Kevin Gurney
>

Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Posted by Kevin Gurney <kg...@mathworks.com>.
It seems like the mailing list stripped out the design doc I attached for some reason.

Here is a link to the same document hosted online instead:

https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4

Note: This link is only a temporary solution (will expire on February 21, 2021). It would be ideal if we could move this to a better place like the Arrow Confluence Design Documents area.

Thanks,

Kevin
________________________________
From: Kevin Gurney <kg...@mathworks.com>
Sent: Thursday, January 21, 2021 4:47 PM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Cc: Jeremy Hughes <jh...@mathworks.com>; Nick Haddad <nh...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La <fi...@mathworks.com>; Tahsin Hassan <th...@mathworks.com>; Yann Debray <yd...@mathworks.com>
Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow

Hello All,

MathWorks is interested in collaborating with the rest of the Arrow community to build out a MATLAB interface to Arrow memory. We envision an interface analogous to the other language bindings, with packaged classes and functions like:

  *   arrow.Array
  *   arrow.TableReader
  *   arrow.type.Float64
  *   ...

In the past, several MathWorkers worked with the Arrow community to develop a proof-of-concept MATLAB interface for reading/writing Feather V1 files by leveraging the Arrow C++ libraries. Since then, the Arrow project has evolved considerably, and we'd like to work with the community to expand MATLAB's ability to interoperate with the broader Arrow ecosystem.

Attached to this email is a lightweight design document which lays out a high-level direction for these development efforts. We welcome any and all feedback on this document.

It would be great to move this design document to some place that is more easily accessible and publicly archived for all members of the Arrow community. At first glance, the Arrow Confluence Design Documents area (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents) seems like the ideal place. However, if you have other suggestions of how best to collaborate on this document, please let me know.

We are excited to work together with the rest of the Arrow community to make this a reality.

Best Regards,

Kevin Gurney