You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "0x26res (via GitHub)" <gi...@apache.org> on 2023/02/17 14:52:03 UTC
[GitHub] [arrow] 0x26res opened a new issue, #34238: Segfault when calling groupby on table with misaligned chunks
0x26res opened a new issue, #34238:
URL: https://github.com/apache/arrow/issues/34238
### Describe the bug, including details regarding any error messages, version, and platform.
- I have a table with ~100k records and 4 chunks of various sizes `[32768, 32768, 29599, 16692]`
- key1 has got high cardinality (30k)
- key2 has got low cardinality (10)
| key1 | key2 | value |
|:-----------|:--------|----------:|
| KEY1_12226 | KEY1_08 | 0.348599 |
| KEY1_10214 | KEY1_08 | 0.954173 |
| KEY1_26821 | KEY1_09 | 0.416615 |
| KEY1_24557 | KEY1_06 | 0.883226 |
| KEY1_27823 | KEY1_08 | 0.0127225 |
I'm trying to do a groupby on key1, key2 to get the sum of value. It works fine in general. But when I preprocess the data I misalign the chunks in my table and it fails.
```
import numpy as np
import pyarrow as pa
KEYS_1 = [f"KEY1_{i:05d}" for i in range(30_000)]
KEYS_2 = [f"KEY1_{i:02d}" for i in range(10)]
SIDE = ["LEFT", "RIGHT"]
def generate_table(sizes):
batches = [
pa.record_batch(
[
np.random.choice(KEYS_1, size),
np.random.choice(KEYS_2, size),
np.random.rand(size),
],
["key1", "key2", "value"],
)
for size in sizes
]
return pa.Table.from_batches(batches)
table = generate_table([32768, 32768, 29599, 16692])
# This works well:
pa.TableGroupBy(table, ["key1", "key2"]).aggregate(
[
["value", "sum"],
]
)
# This misaligns the chunks
table = table.set_column(
table.schema.get_field_index("value"),
"value",
table["value"].combine_chunks(),
)
print("HERE")
pa.TableGroupBy(table, ["key1", "key2"]).aggregate(
[
["value", "sum"],
]
) # segfault :-(
print("NEVER THERE")
```
It took me a while to go to the bottom of the problem. The size of the chunks and the cardinality of the keys seem to play an important factor in weather it fails or not.
The short term solution is for me to call combine_chunks before the groupby.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34238: [C++][Python] Segfault when calling groupby on table with misaligned chunks
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34238:
URL: https://github.com/apache/arrow/issues/34238#issuecomment-1439081019
This is resolved by https://github.com/apache/arrow/pull/14867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed issue #34238: [C++][Python] Segfault when calling groupby on table with misaligned chunks
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche closed issue #34238: [C++][Python] Segfault when calling groupby on table with misaligned chunks
URL: https://github.com/apache/arrow/issues/34238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34238: [C++][Python] Segfault when calling groupby on table with misaligned chunks
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34238:
URL: https://github.com/apache/arrow/issues/34238#issuecomment-1434792719
CC @westonpace, I think there are a number of places currently where Acero doesn't quite handle misaligned buffers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34238: [C++][Python] Segfault when calling groupby on table with misaligned chunks
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34238:
URL: https://github.com/apache/arrow/issues/34238#issuecomment-1435045266
True, though I think we are using two different versions of "align" here.
There is a known issue with "[data alignment](http://www.songho.ca/misc/alignment/dataalign.html)". We require at least 8-byte aligned pointers. There is a PR in the works (https://github.com/apache/arrow/pull/14758) to enforce alignment (requires a data copy) as data goes into Acero.
However, I believe what @0x26res is referring to here is chunk alignment within a table. Generally one assumes that a table consists of consistent chunks, e.g. a vector of record batches. However, it is possible for some columns to be chunked differently than other columns. Acero handles this by requiring all input be record batches, so misaligned tables are aligned by zero-copy slicing into appropriate sized chunks. This defect stems from the fact that group_by was previously implemented outside of Acero using an internal helper function.
TL;DR: This is fixed by https://github.com/apache/arrow/pull/14867 . I ran the given example (thanks for this!) and can reproduce on master but not on #14867.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34238: [C++][Python] Segfault when calling groupby on table with misaligned chunks
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34238:
URL: https://github.com/apache/arrow/issues/34238#issuecomment-1435055776
Ah, I assumed it was data alignment! Thanks for digging into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org