Is sparse matrix in Apache Arrow scope?

[Kouhei Sutou <ko...@clear-code.com>]

Hi,

I have a question.
Is there any plan to support sparse matrix in Apache Arrow?
I just want to know whether sparse matrix is out of scope or
not.


Thanks,
--
kou

Re: Is sparse matrix in Apache Arrow scope?

[Kouhei Sutou <ko...@clear-code.com>]

Hi,

Thanks for sharing your opinion!

In <CA...@mail.gmail.com>
  "Re: Is sparse matrix in Apache Arrow scope?" on Tue, 18 Apr 2017 11:41:48 -0400,
  Wes McKinney <we...@gmail.com> wrote:

> Did you want to propose metadata and a memory layout / data structure for
> these, and we can discuss there?

Ah, no. I just want to introduce Apache Arrow to Ruby
users. I need to know about the current Apache Arrow and the
future Apache Arrow for it.

There are some Ruby libraries that support sparse matrix but
there is no de facto standard library like scipy.sparse. If
Ruby users work together with other language users via
Apache Arrow, Ruby users will be happy. (And I hope other
language users are also happy.)

I'm not a sparse matrix expert yet. So I have no idea that
cares high performance yet. I guess that existing library
such as scipy.sparse based a memory layout / data structure
is better.

Anyway, I opened an issue for discussing about this:
  https://issues.apache.org/jira/browse/ARROW-854


Thanks,
--
kou

Re: Is sparse matrix in Apache Arrow scope?

[Wes McKinney <we...@gmail.com>]

hi Kou -- since we have metadata for tensors/ndarrays and support in C++,
having analogous metadata for sparse matrices seems reasonable. Someone
else asked me about this in person recently (related to machine learning
applications).

While many Arrow implementations will be focused on the in-memory columnar
/ record batch data model, having these traditional scientific computing
memory layouts available for shared memory IPC will open up the Arrow C/C++
libraries to more use cases. We wouldn't necessarily expect other Arrow
implementations (like Java) to implement these, though, unless they needed
them.

Did you want to propose metadata and a memory layout / data structure for
these, and we can discuss there?

Thanks!
Wes

On Tue, Apr 18, 2017 at 1:14 AM, Kouhei Sutou <ko...@clear-code.com> wrote:

> Hi,
>
> I have a question.
> Is there any plan to support sparse matrix in Apache Arrow?
> I just want to know whether sparse matrix is out of scope or
> not.
>
>
> Thanks,
> --
> kou
>