You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Mitar <mm...@gmail.com> on 2019/03/30 22:14:24 UTC

Support for numpy matrix

Hi!

It seems numpy's matrix is not supported in recent versions of pyarrow:

https://issues.apache.org/jira/browse/ARROW-3399

Any ideas why this would be happening?


Mitar

-- 
http://mitar.tnode.com/
https://twitter.com/mitar_m

Re: Support for numpy matrix

Posted by Mitar <mm...@gmail.com>.
Hi!

I agree. This is in fact all information which is already there. :-)


Mitar

On Sat, Mar 30, 2019 at 8:40 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi Mitar,
>
> Let's discuss further on JIRA? It's best to keep all the information
> about the issue in one place.
>
> Thanks
>
> On Sat, Mar 30, 2019 at 7:42 PM Mitar <mm...@gmail.com> wrote:
> >
> > Hi!
> >
> > I added:
> >
> > serialization_context.register_type(
> >     np.matrix, 'np.matrix',
> >     custom_serializer=_serialize_numpy_array_list,
> >     custom_deserializer=_deserialize_numpy_array_list)
> >
> > But it did not help. Probably also because np.matrix is a subclas of
> > np.ndarray anyway. So no change here.
> >
> > An interesting fact is that this worked in older versions of numpy,
> > but stopped in numpy 1.15.2. It works with numpy 1.14.3. So it is them
> > changing something.
> >
> >
> > Mitar
> >
> > On Sat, Mar 30, 2019 at 3:34 PM Philipp Moritz <pc...@gmail.com> wrote:
> > >
> > > Hey Mitar,
> > >
> > > It might be as simple as adding a handler here:
> > > https://github.com/apache/arrow/blob/master/python/pyarrow/serialization.py#L300
> > >
> > > Do you want to try that?
> > >
> > > -- Philipp.
> > >
> > > On Sat, Mar 30, 2019 at 3:22 PM Mitar <mm...@gmail.com> wrote:
> > >
> > > > Hi!
> > > >
> > > > I do not know where to start looking into this? Not sure if I have
> > > > enough knowledge about arrow to be able to make a PR.
> > > >
> > > >
> > > > Miar
> > > >
> > > > On Sat, Mar 30, 2019 at 3:17 PM Wes McKinney <we...@gmail.com> wrote:
> > > > >
> > > > > hi Mitar,
> > > > >
> > > > > I see you reported the issue on October 2 and no one has volunteered
> > > > > to fix it yet. Are you up to submit a PR?
> > > > >
> > > > > Thanks
> > > > > Wes
> > > > >
> > > > > On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
> > > > > >
> > > > > > Hi!
> > > > > >
> > > > > > It seems numpy's matrix is not supported in recent versions of pyarrow:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/ARROW-3399
> > > > > >
> > > > > > Any ideas why this would be happening?
> > > > > >
> > > > > >
> > > > > > Mitar
> > > > > >
> > > > > > --
> > > > > > http://mitar.tnode.com/
> > > > > > https://twitter.com/mitar_m
> > > >
> > > >
> > > >
> > > > --
> > > > http://mitar.tnode.com/
> > > > https://twitter.com/mitar_m
> > > >
> >
> >
> >
> > --
> > http://mitar.tnode.com/
> > https://twitter.com/mitar_m



-- 
http://mitar.tnode.com/
https://twitter.com/mitar_m

Re: Support for numpy matrix

Posted by Wes McKinney <we...@gmail.com>.
hi Mitar,

Let's discuss further on JIRA? It's best to keep all the information
about the issue in one place.

Thanks

On Sat, Mar 30, 2019 at 7:42 PM Mitar <mm...@gmail.com> wrote:
>
> Hi!
>
> I added:
>
> serialization_context.register_type(
>     np.matrix, 'np.matrix',
>     custom_serializer=_serialize_numpy_array_list,
>     custom_deserializer=_deserialize_numpy_array_list)
>
> But it did not help. Probably also because np.matrix is a subclas of
> np.ndarray anyway. So no change here.
>
> An interesting fact is that this worked in older versions of numpy,
> but stopped in numpy 1.15.2. It works with numpy 1.14.3. So it is them
> changing something.
>
>
> Mitar
>
> On Sat, Mar 30, 2019 at 3:34 PM Philipp Moritz <pc...@gmail.com> wrote:
> >
> > Hey Mitar,
> >
> > It might be as simple as adding a handler here:
> > https://github.com/apache/arrow/blob/master/python/pyarrow/serialization.py#L300
> >
> > Do you want to try that?
> >
> > -- Philipp.
> >
> > On Sat, Mar 30, 2019 at 3:22 PM Mitar <mm...@gmail.com> wrote:
> >
> > > Hi!
> > >
> > > I do not know where to start looking into this? Not sure if I have
> > > enough knowledge about arrow to be able to make a PR.
> > >
> > >
> > > Miar
> > >
> > > On Sat, Mar 30, 2019 at 3:17 PM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > hi Mitar,
> > > >
> > > > I see you reported the issue on October 2 and no one has volunteered
> > > > to fix it yet. Are you up to submit a PR?
> > > >
> > > > Thanks
> > > > Wes
> > > >
> > > > On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > It seems numpy's matrix is not supported in recent versions of pyarrow:
> > > > >
> > > > > https://issues.apache.org/jira/browse/ARROW-3399
> > > > >
> > > > > Any ideas why this would be happening?
> > > > >
> > > > >
> > > > > Mitar
> > > > >
> > > > > --
> > > > > http://mitar.tnode.com/
> > > > > https://twitter.com/mitar_m
> > >
> > >
> > >
> > > --
> > > http://mitar.tnode.com/
> > > https://twitter.com/mitar_m
> > >
>
>
>
> --
> http://mitar.tnode.com/
> https://twitter.com/mitar_m

Re: Support for numpy matrix

Posted by Mitar <mm...@gmail.com>.
Hi!

I added:

serialization_context.register_type(
    np.matrix, 'np.matrix',
    custom_serializer=_serialize_numpy_array_list,
    custom_deserializer=_deserialize_numpy_array_list)

But it did not help. Probably also because np.matrix is a subclas of
np.ndarray anyway. So no change here.

An interesting fact is that this worked in older versions of numpy,
but stopped in numpy 1.15.2. It works with numpy 1.14.3. So it is them
changing something.


Mitar

On Sat, Mar 30, 2019 at 3:34 PM Philipp Moritz <pc...@gmail.com> wrote:
>
> Hey Mitar,
>
> It might be as simple as adding a handler here:
> https://github.com/apache/arrow/blob/master/python/pyarrow/serialization.py#L300
>
> Do you want to try that?
>
> -- Philipp.
>
> On Sat, Mar 30, 2019 at 3:22 PM Mitar <mm...@gmail.com> wrote:
>
> > Hi!
> >
> > I do not know where to start looking into this? Not sure if I have
> > enough knowledge about arrow to be able to make a PR.
> >
> >
> > Miar
> >
> > On Sat, Mar 30, 2019 at 3:17 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > hi Mitar,
> > >
> > > I see you reported the issue on October 2 and no one has volunteered
> > > to fix it yet. Are you up to submit a PR?
> > >
> > > Thanks
> > > Wes
> > >
> > > On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
> > > >
> > > > Hi!
> > > >
> > > > It seems numpy's matrix is not supported in recent versions of pyarrow:
> > > >
> > > > https://issues.apache.org/jira/browse/ARROW-3399
> > > >
> > > > Any ideas why this would be happening?
> > > >
> > > >
> > > > Mitar
> > > >
> > > > --
> > > > http://mitar.tnode.com/
> > > > https://twitter.com/mitar_m
> >
> >
> >
> > --
> > http://mitar.tnode.com/
> > https://twitter.com/mitar_m
> >



-- 
http://mitar.tnode.com/
https://twitter.com/mitar_m

Re: Support for numpy matrix

Posted by Philipp Moritz <pc...@gmail.com>.
Hey Mitar,

It might be as simple as adding a handler here:
https://github.com/apache/arrow/blob/master/python/pyarrow/serialization.py#L300

Do you want to try that?

-- Philipp.

On Sat, Mar 30, 2019 at 3:22 PM Mitar <mm...@gmail.com> wrote:

> Hi!
>
> I do not know where to start looking into this? Not sure if I have
> enough knowledge about arrow to be able to make a PR.
>
>
> Miar
>
> On Sat, Mar 30, 2019 at 3:17 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > hi Mitar,
> >
> > I see you reported the issue on October 2 and no one has volunteered
> > to fix it yet. Are you up to submit a PR?
> >
> > Thanks
> > Wes
> >
> > On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
> > >
> > > Hi!
> > >
> > > It seems numpy's matrix is not supported in recent versions of pyarrow:
> > >
> > > https://issues.apache.org/jira/browse/ARROW-3399
> > >
> > > Any ideas why this would be happening?
> > >
> > >
> > > Mitar
> > >
> > > --
> > > http://mitar.tnode.com/
> > > https://twitter.com/mitar_m
>
>
>
> --
> http://mitar.tnode.com/
> https://twitter.com/mitar_m
>

Re: Support for numpy matrix

Posted by Wes McKinney <we...@gmail.com>.
Let's discuss on the JIRA issue. AFAIK we have no unit tests relating
to numpy.matrix so most likely this was an accidental "feature". If
there is no other volunteer to help on the issue it's hard to predict
when it will get done. numpy.matrix isn't used by very many people

On Sat, Mar 30, 2019 at 5:22 PM Mitar <mm...@gmail.com> wrote:
>
> Hi!
>
> I do not know where to start looking into this? Not sure if I have
> enough knowledge about arrow to be able to make a PR.
>
>
> Miar
>
> On Sat, Mar 30, 2019 at 3:17 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > hi Mitar,
> >
> > I see you reported the issue on October 2 and no one has volunteered
> > to fix it yet. Are you up to submit a PR?
> >
> > Thanks
> > Wes
> >
> > On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
> > >
> > > Hi!
> > >
> > > It seems numpy's matrix is not supported in recent versions of pyarrow:
> > >
> > > https://issues.apache.org/jira/browse/ARROW-3399
> > >
> > > Any ideas why this would be happening?
> > >
> > >
> > > Mitar
> > >
> > > --
> > > http://mitar.tnode.com/
> > > https://twitter.com/mitar_m
>
>
>
> --
> http://mitar.tnode.com/
> https://twitter.com/mitar_m

Re: Support for numpy matrix

Posted by Mitar <mm...@gmail.com>.
Hi!

I do not know where to start looking into this? Not sure if I have
enough knowledge about arrow to be able to make a PR.


Miar

On Sat, Mar 30, 2019 at 3:17 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi Mitar,
>
> I see you reported the issue on October 2 and no one has volunteered
> to fix it yet. Are you up to submit a PR?
>
> Thanks
> Wes
>
> On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
> >
> > Hi!
> >
> > It seems numpy's matrix is not supported in recent versions of pyarrow:
> >
> > https://issues.apache.org/jira/browse/ARROW-3399
> >
> > Any ideas why this would be happening?
> >
> >
> > Mitar
> >
> > --
> > http://mitar.tnode.com/
> > https://twitter.com/mitar_m



-- 
http://mitar.tnode.com/
https://twitter.com/mitar_m

Re: Support for numpy matrix

Posted by Wes McKinney <we...@gmail.com>.
hi Mitar,

I see you reported the issue on October 2 and no one has volunteered
to fix it yet. Are you up to submit a PR?

Thanks
Wes

On Sat, Mar 30, 2019 at 5:14 PM Mitar <mm...@gmail.com> wrote:
>
> Hi!
>
> It seems numpy's matrix is not supported in recent versions of pyarrow:
>
> https://issues.apache.org/jira/browse/ARROW-3399
>
> Any ideas why this would be happening?
>
>
> Mitar
>
> --
> http://mitar.tnode.com/
> https://twitter.com/mitar_m