You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2024/04/27 13:15:19 UTC
[I] Display of built in window functions do not work with struct elements [datafusion-python]
timsaucer opened a new issue, #647:
URL: https://github.com/apache/datafusion-python/issues/647
**Describe the bug**
When attempting to call `show()` on a DataFrame that contains a built in window function on a column that has struct elements, it produces the error `Compute error: concat requires input of at least one array`. However other functions such as `count()` do not have issues. I have less experience with DataFusion, so I just expected `count()` to do a full evaluation like it does in pyspark, so it's possible that my assumption is incorrect in that having any bearing on this error.
**To Reproduce**
This minimal example can reproduce the window function working properly on a simple element type and failing with a very simple struct.
```
import pyarrow as pa
from datafusion import SessionContext
import datafusion.functions as F
# taken from datafusion/tests/test_dataframe.py
def struct_df():
ctx = SessionContext()
# create a RecordBatch and a new DataFrame from it
batch = pa.RecordBatch.from_arrays(
[pa.array([{"c": 1}, {"c": 2}, {"c": 3}]), pa.array([4, 5, 6])],
names=["a", "b"],
)
return ctx.create_dataframe([[batch]])
df = struct_df()
df.show()
df.select(F.col("a"), F.col("b"), F.window("lag", [F.col("b")]).alias("lag_b")).show()
print("Calling count on lag a: ", df.select(F.col("a"), F.col("b"), F.window("lag", [F.col("a")]).alias("lag_a")).count())
df.select(F.col("a"), F.col("b"), F.window("lag", [F.col("a")]).alias("lag_a")).show()
```
Produces the following output:
```
DataFrame()
+--------+---+
| a | b |
+--------+---+
| {c: 1} | 4 |
| {c: 2} | 5 |
| {c: 3} | 6 |
+--------+---+
DataFrame()
+--------+---+-------+
| a | b | lag_b |
+--------+---+-------+
| {c: 1} | 4 | |
| {c: 2} | 5 | 4 |
| {c: 3} | 6 | 5 |
+--------+---+-------+
Calling count on lag a: 3
Traceback (most recent call last):
File "/Users/tsaucer/src/arrow-datafusion-python/example_lag_struct.py", line 25, in <module>
df.select(F.col("a"), F.col("b"), F.window("lag", [F.col("a")]).alias("lag_a")).show()
Exception: Arrow error: Compute error: concat requires input of at least one array
```
In searching the web there was a similar error thrown that this old MR resolved in sort operations: https://github.com/apache/arrow/pull/9275/files#diff-3ee8e6ac2472badc7bb448c360f56ed60f06a787d1f45ea589d9e213eaf2ae82
**Expected behavior**
Calling `show()` on a window function with a struct column type should operate similar to simple column types.
**Additional context**
I'm willing to work on this myself, but I'm not familiar with the internals of the plan execution. I've looked around myself to see if I can find anything obvious, but nothing is jumping out at me. If you can provide any directions or pointers, I would appreciate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Display of built in window functions do not work with struct elements [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #647: Display of built in window functions do not work with struct elements
URL: https://github.com/apache/datafusion-python/issues/647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Display of built in window functions do not work with struct elements [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #647:
URL: https://github.com/apache/datafusion-python/issues/647#issuecomment-2088441787
Closing in favor of https://github.com/apache/datafusion/issues/10328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org