You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "rustyconover (via GitHub)" <gi...@apache.org> on 2024/09/22 16:16:41 UTC
[I] [Python] run_end_encode doesn't support structs and lists [arrow]
rustyconover opened a new issue, #44183:
URL: https://github.com/apache/arrow/issues/44183
### Describe the bug, including details regarding any error messages, version, and platform.
When attempting to use a `RunEndEncoded` array with either a `struct` or a `list`, an exception is raised indicating that no matching kernel is available.
#### Steps to Reproduce:
Please run the following example code to reproduce the issue:
```python
import pyarrow as pa
for data_type, values in [
[pa.struct([pa.field("age", pa.int32())]), {"age": 20}],
[pa.list_(pa.int32()), [20]]
]:
try:
schema = pa.schema([pa.field("data", pa.run_end_encoded(pa.int16(), data_type))])
data = [
{"data": values},
]
table = pa.Table.from_pylist(data, schema=schema)
except Exception as e:
print(f"Failed with {data_type} {e}")
```
#### Observed Output:
```
Failed with struct<age: int32> Function 'run_end_encode' has no kernel matching input types (struct<age: int32>)
Failed with list<item: int32> Function 'run_end_encode' has no kernel matching input types (list<item: int32>)
```
#### Expected Behavior:
The code should correctly create a `RunEndEncoded` array using both `struct` and `list` types without raising exceptions.
#### Environment:
- PyArrow version: 18.0.0.dev
- Python version: 3.12.5
- Platform: Mac OS X
#### Additional Context:
The failure seems to suggest that the `run_end_encode` function does not currently support `struct` or `list` types, but it's not explicitly documented whether this is intentional or an oversight.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-2369638060
> Why wouldn't it bring great compression rates if the developer knows the column is mostly constant values?
All it takes is a new random or misaligned column (struct field) to mess up the repetitiveness of the data.
If you *know* the data is mostly constant values, you don't need `run_end_encode`, because you can produce the run-end encoded array directly without comparing the struct values.
You can also go for a struct of run-end-encoded fields (not all of them have to be run-end-encoded) and if the whole struct repeats you can share the same `run_ends` array among the fields (no copying needed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-2368675703
Comparing nested types for equality to run-end encode them can be expensive and unlikely to bring good compression rates. Run-end encoding works better on flat columns.
That said, the most likely reason for a kernel not supporting REE arrays yet is usually: it hasn't been implemented. Because, you know, it takes time ($) to implement custom kernels for REE arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "BenMagyar (via GitHub)" <gi...@apache.org>.
BenMagyar commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-5131534412
I can!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-2367158401
cc @felipecrv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #44183: [Python] run_end_encode doesn't support structs and lists
URL: https://github.com/apache/arrow/issues/44183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-5128701340
Issue resolved by pull request 50534
https://github.com/apache/arrow/pull/50534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-5128709343
@BenMagyar can you post here so that we can assign this issue to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "jagill (via GitHub)" <gi...@apache.org>.
jagill commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-2594286847
As a data point, I am running into this in production, from sources I don't control. So maybe they don't know it's not the best compression :).
That doesn't change the fact that it takes effort to implement, of course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] run_end_encode doesn't support structs and lists [arrow]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on issue #44183:
URL: https://github.com/apache/arrow/issues/44183#issuecomment-2368931985
Hi @felipecrv
Why wouldn't it bring great compression rates if the developer knows the column is mostly constant values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org