You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "anemohan (via GitHub)" <gi...@apache.org> on 2023/12/11 15:58:47 UTC
[I] ArrowNotImplementedError('Unsupported cast from list> to utf8 using function cast_string') [arrow]
anemohan opened a new issue, #39172:
URL: https://github.com/apache/arrow/issues/39172
### Describe the usage question you have. Please include as many useful details as possible.
Hello, I use a schema file defining all the column types. I have a column with a list of dicts and when I try to call the dataset with this schema I get an error stating:
```
The original error is below:
------------------------
ArrowNotImplementedError('Unsupported cast from list<item: struct<count: int64, gender: string, name: string, probability: double>> to utf8 using function cast_string')
Traceback:
---------
File "/opt/conda/lib/python3.10/site-packages/dask/dataframe/utils.py", line 193, in raise_on_meta_error
yield
File "/opt/conda/lib/python3.10/site-packages/dask/dataframe/core.py", line 6897, in _emulate
return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
File "/opt/conda/lib/python3.10/site-packages/intakewrapper/MHParquetSource.py", line 67, in __call__
df = pd.read_parquet(
File "/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py", line 509, in read_parquet
return impl.read(
File "/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py", line 227, in read
pa_table = self.api.parquet.read_table(
File "/opt/conda/lib/python3.10/site-packages/pyarrow/parquet/__init__.py", line 2780, in read_table
return dataset.read(columns=columns, use_threads=use_threads,
File "/opt/conda/lib/python3.10/site-packages/pyarrow/parquet/__init__.py", line 2443, in read
table = self._dataset.to_table(
File "pyarrow/_dataset.pyx", line 304, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 2549, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
```
Can someone help me with this?
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArrowNotImplementedError('Unsupported cast from list> to utf8 using function cast_string') [arrow]
Posted by "anemohan (via GitHub)" <gi...@apache.org>.
anemohan commented on issue #39172:
URL: https://github.com/apache/arrow/issues/39172#issuecomment-1850697056
Ok, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArrowNotImplementedError('Unsupported cast from list> to utf8 using function cast_string') [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39172:
URL: https://github.com/apache/arrow/issues/39172#issuecomment-1850524584
It seems that one of the columns in your dataset has a list type, but you specified it in the schema as a string column. When the types don't match, we try to cast the column after reading to the type in the schema, but a cast from list type to string is not supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ArrowNotImplementedError('Unsupported cast from list> to utf8 using function cast_string') [arrow]
Posted by "anemohan (via GitHub)" <gi...@apache.org>.
anemohan closed issue #39172: ArrowNotImplementedError('Unsupported cast from list<item: struct<count: int64, gender: string, name: string, probability: double>> to utf8 using function cast_string')
URL: https://github.com/apache/arrow/issues/39172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org