You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "dlovell (via GitHub)" <gi...@apache.org> on 2023/11/30 12:20:17 UTC
[I] to_*() on a dataframe with a struct col fails with "ArrowInvalid: Schema at index 0 was different" [arrow-datafusion-python]
dlovell opened a new issue, #541:
URL: https://github.com/apache/arrow-datafusion-python/issues/541
**Describe the bug**
calling `to_*` on a dataframe with a `struct` column fails unless all the struct fields are of type `string`
**To Reproduce**
Steps to reproduce the behavior:
```
import pandas as pd
from datafusion import (
column,
functions as f,
SessionContext,
)
def make_df(n=30):
return pd.DataFrame(
{
"a": pd.date_range(start="2020-01-01", freq="M", periods=n),
"b": range(n),
"c": pd.Series(range(n)).add(0.1),
"d": pd.Series(range(n)).astype(str),
}
)
# ).astype(str)
# if all struct fields are str type, the failure does not occur
ctx = SessionContext()
t = ctx.from_pandas(make_df(), "t").select(
column("a"),
f.functions.struct(column("b"), column("c"), column("d")).alias("bcd"),
)
# this fails, as do all invocations of to_* methods
t.to_pandas()
```
**Expected behavior**
I would expect no failure to occur, as is the case if you first cast all the data to type `str`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] to_*() on a dataframe with a struct col fails with "ArrowInvalid: Schema at index 0 was different" [arrow-datafusion-python]
Posted by "dlovell (via GitHub)" <gi...@apache.org>.
dlovell closed issue #541: to_*() on a dataframe with a struct col fails with "ArrowInvalid: Schema at index 0 was different"
URL: https://github.com/apache/arrow-datafusion-python/issues/541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] to_*() on a dataframe with a struct col fails with "ArrowInvalid: Schema at index 0 was different" [arrow-datafusion-python]
Posted by "dlovell (via GitHub)" <gi...@apache.org>.
dlovell commented on issue #541:
URL: https://github.com/apache/arrow-datafusion-python/issues/541#issuecomment-1875324974
This is fixed in `34.0.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] to_*() on a dataframe with a struct col fails with "ArrowInvalid: Schema at index 0 was different" [arrow-datafusion-python]
Posted by "iammax (via GitHub)" <gi...@apache.org>.
iammax commented on issue #541:
URL: https://github.com/apache/arrow-datafusion-python/issues/541#issuecomment-1849969601
I was having a similar issue with version 31.0.0. Small example snip:
```
import datafusion as dfu
ctx = dfu.SessionContext()
ctx.register_csv('delta', 'test.csv')
result = ctx.sql('SELECT col1, COUNT(DISTINCT col2) FROM delta GROUP BY col1')
```
This assigns result to a ```datafusion.Dataframe``` object as expected. I can see it has the correct values in it by printing it out in terminal/jupyter. However, if I do ```result.to_polars()``` (or to_anything else) I get the same error as the original post.
```
ArrowInvalid: Schema at index 0 was different:
col1: int64
COUNT(DISTINCT delta.col2): int64
vs
delta.col1: int64
COUNT(DISTINCT delta.col2): int64
```
However it works in version 33.0,0 (I think that's the current version), so I assume there was a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] to_*() on a dataframe with a struct col fails with "ArrowInvalid: Schema at index 0 was different" [arrow-datafusion-python]
Posted by "dlovell (via GitHub)" <gi...@apache.org>.
dlovell commented on issue #541:
URL: https://github.com/apache/arrow-datafusion-python/issues/541#issuecomment-1841260108
the underlying issue is probably https://github.com/apache/arrow-datafusion/issues/8118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org