You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ladislav-dolezal (via GitHub)" <gi...@apache.org> on 2024/09/06 09:32:58 UTC
[I] ArrowInvalid Error when converting DataFrame to pandas [datafusion]
ladislav-dolezal opened a new issue, #12354:
URL: https://github.com/apache/datafusion/issues/12354
### Describe the bug
I am testing the example from [sql-to-pandas.py](https://github.com/apache/datafusion-python/blob/main/examples/sql-to-pandas.py) and I got en error ArrowInvalid with different schema on the line: pandas_df = df.to_pandas().
ArrowInvalid: Schema at index 0 was different:
passenger_count: int64
trip_count: int32 not null
vs
passenger_count: int64
trip_count: int32
Is there any way how to define trip_acount as not nullable when running select or how to make pandas to accept such a DataFrame.
I am testing with conda env with python 3.11.9 on windows. Pip packages I have listed bellow.
```
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[11], [line 19](vscode-notebook-cell:?execution_count=11&line=19)
[17](vscode-notebook-cell:?execution_count=11&line=17) print(df.schema())
[18](vscode-notebook-cell:?execution_count=11&line=18) # convert to Pandas
---> [19](vscode-notebook-cell:?execution_count=11&line=19) pandas_df = df.to_pandas()
[21](vscode-notebook-cell:?execution_count=11&line=21) # create a chart
[22](vscode-notebook-cell:?execution_count=11&line=22) fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\datafusion\dataframe.py:472, in DataFrame.to_pandas(self)
[466](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:466) def to_pandas(self) -> pd.DataFrame:
[467](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:467) """Execute the :py:class:`DataFrame` and convert it into a Pandas DataFrame.
[468](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:468)
[469](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:469) Returns:
[470](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:470) Pandas DataFrame.
[471](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:471) """
--> [472](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:472) return self.df.to_pandas()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\table.pxi:4865, in pyarrow.lib.Table.from_batches()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\error.pxi:92, in pyarrow.lib.check_status()
ArrowInvalid: Schema at index 0 was different:
passenger_count: int64
trip_count: int32 not null
vs
passenger_count: int64
trip_count: int32
```
Data source for the data is here:
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet
### To Reproduce
```
from datafusion import SessionContext
ctx = SessionContext()
# Register table with context
ctx.register_parquet('taxi', 'datasets/yellow_tripdata_2024-01.parquet')
# Execute SQL
df = ctx.sql("select passenger_count, cast(count(*) as int) as trip_count "
"from taxi "
"where passenger_count is not null "
"group by passenger_count "
"order by passenger_count")
# convert to Pandas
df.show()
print(df.schema())
# convert to Pandas
pandas_df = df.to_pandas()
# create a chart
fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
fig.savefig('chart.png')
```
### Expected behavior
_No response_
### Additional context
Package Version
----------------- -----------
asttokens 2.0.5
colorama 0.4.6
comm 0.2.1
datafusion 40.1.0
debugpy 1.6.7
decorator 5.1.1
executing 0.8.3
ipykernel 6.28.0
ipython 8.25.0
jedi 0.19.1
jupyter_client 8.6.0
jupyter_core 5.7.2
matplotlib-inline 0.1.6
nest-asyncio 1.6.0
numpy 2.1.1
packaging 24.1
pandas 2.2.2
parso 0.8.3
pip 24.2
platformdirs 3.10.0
prompt-toolkit 3.0.43
psutil 5.9.0
pure-eval 0.2.2
pyarrow 17.0.0
Pygments 2.15.1
python-dateutil 2.9.0.post0
pytz 2024.1
pywin32 305.1
pyzmq 25.1.2
setuptools 72.1.0
six 1.16.0
stack-data 0.2.0
tornado 6.4.1
traitlets 5.14.3
typing_extensions 4.11.0
tzdata 2024.1
wcwidth 0.2.5
wheel 0.43.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org