You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "hinxx (via GitHub)" <gi...@apache.org> on 2023/03/07 13:44:02 UTC
[GitHub] [arrow] hinxx opened a new issue, #34486: Converting arrow sparse_union table to pandas dataframe
hinxx opened a new issue, #34486:
URL: https://github.com/apache/arrow/issues/34486
### Describe the bug, including details regarding any error messages, version, and platform.
I can not seem to convert pyarrow sparse_union table to pandas dataframe:
```
>>> table
pyarrow.Table
name: string
timestamp: timestamp[ns]
value: sparse_union<_union_0: int64=0, _union_1: double=1, _union_2: string=2>
child 0, _union_0: int64
child 1, _union_1: double
child 2, _union_2: string
----
name: [["row1","row2","row3","row4","row5","row6"]]
timestamp: [[2022-07-15 12:40:13.439549952,2022-07-15 12:40:13.439546880,2023-02-08 09:13:32.287076352,2023-02-08 09:13:32.587076352,2022-07-07 14:23:10.092787968,2022-07-15 12:40:13.839546624]]
value: [ -- is_valid: all not null -- type_ids: [1,1,0,0,2,1]
-- child 0 type: int64
[null,null,5,53,null,null]
-- child 1 type: double
[0.011021,-32580,null,null,null,-33580]
-- child 2 type: string
[null,null,null,null,"3.10.0",null]]
>>> df=table.to_pandas()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow/array.pxi", line 835, in pyarrow.lib._PandasConvertible.to_pandas
return self._to_pandas(options, categories=categories,
File "pyarrow/table.pxi", line 4087, in pyarrow.lib.Table._to_pandas
mgr = table_to_blockmanager(
File "/arrow/python/pyarrow/pandas_compat.py", line 820, in table_to_blockmanager
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
File "/arrow/python/pyarrow/pandas_compat.py", line 1169, in _table_to_blocks
result = pa.lib.table_to_blocks(options, block_table, categories,
File "pyarrow/table.pxi", line 2743, in pyarrow.lib.table_to_blocks
check_status(
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
raise ArrowNotImplementedError(message)
pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of type sparse_union<_union_0: int64=0, _union_1: double=1, _union_2: string=2> is known.
```
Looks like it is not supported/implemented; is there a workaround/solution for this?
### Component(s)
Integration, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34486: Converting arrow sparse_union table to pandas dataframe
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34486:
URL: https://github.com/apache/arrow/issues/34486#issuecomment-1458940075
What behavior do you want? Does pandas have any kind of union support? You should be able to extract the union into one-mostly-null-column-per-variant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] hinxx closed issue #34486: Converting arrow sparse_union table to pandas dataframe
Posted by "hinxx (via GitHub)" <gi...@apache.org>.
hinxx closed issue #34486: Converting arrow sparse_union table to pandas dataframe
URL: https://github.com/apache/arrow/issues/34486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] hinxx commented on issue #34486: Converting arrow sparse_union table to pandas dataframe
Posted by "hinxx (via GitHub)" <gi...@apache.org>.
hinxx commented on issue #34486:
URL: https://github.com/apache/arrow/issues/34486#issuecomment-1486340417
That was a silly question on my part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org