You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "TPDeramus (via GitHub)" <gi...@apache.org> on 2023/11/16 14:46:57 UTC
[I] [Python] Pyarrow Group_by ChunkedArray and most frequent value [arrow]
TPDeramus opened a new issue, #38747:
URL: https://github.com/apache/arrow/issues/38747
### Describe the usage question you have. Please include as many useful details as possible.
I am so sorry to bug you for something minor, but I can't seem to locate it in the documentation.
I have some data I'm reading in as part of a `parquet` dataset that includes a chunked array (https://arrow.apache.org/docs/python/generated/pyarrow.ChunkedArray.html), which looks something like the following:
`table = pa.table({'IDs': ["A", "A", "A", "B", "B", "C", "C", "C", "C", "D", "D"], 'ChunkedArray':[[7,0,5,8,0],[7,9,1,9,4],[7,1,0,10,7],[8,3,4,0,5],[5,4,5,6,2],[0,5,5,1,6],[8,9,0,10,7],[0,5,10,5,10],[9,6,9,5,10],[8,7,7,3,6],[5,4,10,2,5]]})`
I am trying to apply the `group_by` function to the arrays to get arrays of the `maximum`, `mean`, and most frequent (`mode`) values of each chunked array by ID.
Along the lines of:
```
IDs ChunkedArray
A [7,9,5,10,7]
B [8,4,5,6,5]
C [9,9,10,10,10]
D [8,7,10,3,6]
IDs ChunkedArray
A [7,3.333333333,2,9,3.666666667]
B [6.5,3.5,4.5,3,3.5]
C [4.25,6.25,6,5.25,8.25]
D [6.5,5.5,8.5,2.5,5.5]
IDs ChunkedArray
A [7,NaN,NaN,NaN,NaN]
B [NaN,NaN,NaN,NaN,NaN]
C [0,5,NaN,5,10]
D [NaN,NaN,NaN,NaN,NaN]
```
They can be either separate tables, or all part of the same table or dataframe (whichever is easier to implement), or have their own columns, but the actual dataset is extremely large so I was concerned this might lead to memory read issues.
Calling `aggregate` on the Chunked arrays directly throws errors because the data is a list:
```
>>> table.group_by("IDs").aggregate([("ChunkedArray", "mean")])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow/table.pxi", line 5320, in pyarrow.lib.TableGroupBy.aggregate
File "/usr/Python/mambaforge/lib/python3.10/site-packages/pyarrow/acero.py", line 308, in _group_by
return decl.to_table(use_threads=True)
File "pyarrow/_acero.pyx", line 511, in pyarrow._acero.Declaration.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'hash_mean' has no kernel matching input types (list<item: int64>, uint32)
```
And all my previous attempts at applying flatten on the ChunkedArray before or after calling the functions of interest had not been successful until I tried the result from the following (https://stackoverflow.com/questions/77489901/apply-an-operation-function-to-a-chunkedarray-in-pyarrow):
```
columns = {f'c{i}': pc.list_element(table['ChunkedArray'], i) for i in range(5)}
flat_table = pa.table(columns | {'IDs': table['IDs']})
flat_table.group_by('IDs').aggregate([('c0', 'mean')])
```
However, I'm still short on understanding 2 things.
1. How to apply a function to multiple columns at once in `group_by` and `aggregate` in one go.
2. How I could get the most frequent value, or `mode` of the data using this approach.
I see that `aggregate` outside of `group_by` has the mode function:
https://arrow.apache.org/docs/python/generated/pyarrow.compute.mode.html#pyarrow.compute.mode
But within `table.group_by`:
https://arrow.apache.org/docs/python/compute.html#grouped-aggregations
I see that there are ways to make custom functions:
https://arrow.apache.org/docs/python/compute.html#user-defined-functions
I see that has been discussed previously on custom functions:
https://github.com/apache/arrow/issues/14860
But this is technically built in outside of `group_by`.
Would appreciate any suggestions you are willing to provide.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Pyarrow Group_by ChunkedArray and most frequent value [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38747:
URL: https://github.com/apache/arrow/issues/38747#issuecomment-3547061505
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Pyarrow Group_by ChunkedArray and most frequent value [arrow]
Posted by "TPDeramus (via GitHub)" <gi...@apache.org>.
TPDeramus commented on issue #38747:
URL: https://github.com/apache/arrow/issues/38747#issuecomment-1815040712
Managed to solve issue 1., though probably not as elegantly as I could have, by using the following to generate a list of lists with the summary variable(s):
```
grenerator = [([x for x in table.schema.names if "c" in x][i], "mean") for i in range(len([x for x in table.schema.names if "c" in x]))]
table.group_by("IDs").aggregate(grenerator)
```
The second part however, still eludes me.
Would I be better off filtering by ID and then running the `mode` call individually and appending?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Pyarrow Group_by ChunkedArray and most frequent value [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #38747: [Python] Pyarrow Group_by ChunkedArray and most frequent value
URL: https://github.com/apache/arrow/issues/38747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org