You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Marcel Ackermann (JIRA)" <ji...@apache.org> on 2019/08/13 12:42:00 UTC
[jira] [Updated] (ARROW-6222) Serialising numpy array yields
`pyarrow.lib.ArrowNotImplementedError: list`
[ https://issues.apache.org/jira/browse/ARROW-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcel Ackermann updated ARROW-6222:
------------------------------------
Description:
I want to serialize pytorch tensors, but as they are not implemented in arrow yet I convert them to a numpy array like this: {{t.numpy()}} ([https://pytorch.org/docs/stable/tensors.html?highlight=numpy#torch.Tensor.numpy)] which returns an {{ndarray{{. My tensors are 1-dimensional, the result is a 1-dimensional ndarray.
Calling {{df.to_feather("fname.feather")}} yields {{pyarrow.lib.ArrowNotImplementedError: list<item: float>}}.
Next I tried {{pyarrow.array(t.numpy())}} which results in {{pyarrow.lib.ArrowInvalid: ('Could not convert [\n 0.00500498,\n -0.00732583,\n... with type pyarrow.lib.FloatArray: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column 0 with type object')}}.
I would appreciate if this would work more out-of-the-box.
was:
I want to serialize pytorch tensors, but as they are not implemented in arrow yet I convert them to a numpy array like this: `t.numpy()` ([https://pytorch.org/docs/stable/tensors.html?highlight=numpy#torch.Tensor.numpy)] which returns an `ndarray`. My tensors are 1-dimensional, the result is a 1-dimensional ndarray.
Calling `df.to_feather("fname.feather")` yields `pyarrow.lib.ArrowNotImplementedError: list<item: float>`.
Next I tried `pyarrow.array(t.numpy())` which results in `pyarrow.lib.ArrowInvalid: ('Could not convert [\n 0.00500498,\n -0.00732583,\n... with type pyarrow.lib.FloatArray: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column 0 with type object')`.
I would appreciate if this would work more out-of-the-box.
> Serialising numpy array yields `pyarrow.lib.ArrowNotImplementedError: list<item: float>`
> ----------------------------------------------------------------------------------------
>
> Key: ARROW-6222
> URL: https://issues.apache.org/jira/browse/ARROW-6222
> Project: Apache Arrow
> Issue Type: Bug
> Affects Versions: 0.14.1
> Reporter: Marcel Ackermann
> Priority: Major
>
> I want to serialize pytorch tensors, but as they are not implemented in arrow yet I convert them to a numpy array like this: {{t.numpy()}} ([https://pytorch.org/docs/stable/tensors.html?highlight=numpy#torch.Tensor.numpy)] which returns an {{ndarray{{. My tensors are 1-dimensional, the result is a 1-dimensional ndarray.
> Calling {{df.to_feather("fname.feather")}} yields {{pyarrow.lib.ArrowNotImplementedError: list<item: float>}}.
> Next I tried {{pyarrow.array(t.numpy())}} which results in {{pyarrow.lib.ArrowInvalid: ('Could not convert [\n 0.00500498,\n -0.00732583,\n... with type pyarrow.lib.FloatArray: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column 0 with type object')}}.
> I would appreciate if this would work more out-of-the-box.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)