You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "stinodego (via GitHub)" <gi...@apache.org> on 2024/02/19 11:09:17 UTC
[I] Casting an empty dictionary arrow loses category information [arrow]
stinodego opened a new issue, #40128:
URL: https://github.com/apache/arrow/issues/40128
### Describe the bug, including details regarding any error messages, version, and platform.
Casting a DictionaryArray to another dictionary type will preserve the original dictionary, unless the array is empty.
### Minimal example
```python
import pyarrow as pa
arr = pa.array(["foo", "bar"], pa.dictionary(pa.int32(), pa.string()))
arr_empty = arr[:0]
arr_empty_cast = arr_empty.cast(pa.dictionary(pa.int64(), pa.string()))
print(arr_empty_cast.dictionary) # []
```
### Additional examples
```python
import pyarrow as pa
arr = pa.array(["foo", "bar"], pa.dictionary(pa.int32(), pa.string()))
print(arr.dictionary) # ["foo", "bar"]
arr_short_cast = arr[:1].cast(pa.dictionary(pa.int64(), pa.string()))
print(arr_short_cast.dictionary) # ["foo", "bar"]
arr_empty = arr[:0]
print(arr_empty.dictionary) # ["foo", "bar"]
arr_empty_cast = arr_empty.cast(pa.dictionary(pa.int64(), pa.string()))
print(arr_empty_cast.dictionary) # []
```
### Expected output
The original categories should be preserved when changing the index type, even if the array is empty.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "stinodego (via GitHub)" <gi...@apache.org>.
stinodego commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953888244
> Why do we need to do this? Could you explain your use case?
For Polars, we use `pyarrow` for the converting our data to `pandas`. Our `Enum` data type is similar to a pyarrow Dictionary type with a set number of categories. This converts into a pandas categorical type which also has a set number of categories.
In this conversion, we have to cast the index type from `UInt32` (our default) to `Int64` since pandas does not support unsigned indices. If the data is empty, this cast now loses the category information, and data type after the conversion is wrong.
See the original issue opened in our repo: https://github.com/pola-rs/polars/issues/14582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "Wainberg (via GitHub)" <gi...@apache.org>.
Wainberg commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1961481522
Workaround in the meantime:
```python
import pyarrow as pa
arr = pa.array(["foo", "bar"], pa.dictionary(pa.int32(), pa.string()))
arr_empty = arr[:0]
arr_empty_cast = pa.DictionaryArray.from_arrays(pa.array([], type=pa.int64()), arr_empty.dictionary)
print(arr_empty_cast.dictionary) # ['foo', 'bar']
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1954134119
> We might want to generally skip that fast-path in case output_type_ is a dictionary type?
Ideally we would also do that for nested types that have a dictionary field somewhere.
But we can start with your idea and see what happens (not all scalar kernels might like zero-length inputs?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "stinodego (via GitHub)" <gi...@apache.org>.
stinodego commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953892555
I'm not familiar with the Arrow code base, but I suppose I could give it a shot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953969098
Agreed that we should be consistent for empty arrays.
Haven't tested this, but it might be caused by shortcutting calling the actual kernel for empty input here:
https://github.com/apache/arrow/blob/4dc3d04ae84d97d02443c0cef555a46535925c2b/cpp/src/arrow/compute/exec.cc#L781-L795
This creates a generic array of length 0 of the output type, but so in case of a dictionary output type, it will never have an actual dictionary attached to that return value.
We might want to generally skip that fast-path in case `output_type_` is a dictionary type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953164421
Why do we need to do this? Could you explain your use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953276890
@pitrou Do you have any opinion for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1954197598
We do:
```python
>>> a = pa.array([1,2,3,1]).dictionary_encode()
>>> pc.coalesce(a[:1])
<pyarrow.lib.DictionaryArray object at 0x7f0ef4948ba0>
-- dictionary:
[
1,
2,
3
]
-- indices:
[
0
]
>>> pc.coalesce(a[:0])
<pyarrow.lib.DictionaryArray object at 0x7f0ef49d70d0>
-- dictionary:
[]
-- indices:
[]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1954354911
My take here is that dictionaries are for compression and might be aggressively further-compressed by some operations [1], but `cast` is special and *should preserve* the original dictionary because the goal of a cast is to change only the encoding of the data with as little as possible impact on its semantic properties.
I would keep the fast path that and make only cast kernels the exception. I don't know exactly how without looking at the code.
My reasoning: we don't want code accidentally relying on a guarantee that is stronger than "the dictionary contains the values that are referenced at least once". The path to full determinism here is probably being more aggressive in removing values, than preserving them.
[1] many operations act on the `array[0..length)` values and might gather only the dictionary values that have references to them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1954172837
I am not sure if we actually have many scalar kernels apart from `cast` that would work for dictionary ànd return dictionary?
Checking some non-scalar (vector) kernels, those seem to preserve the dictionary correctly for empty input (using `arr` from the top post):
```
In [8]: pc.filter(arr[:0], pa.array([], pa.bool_()))
Out[8]:
<pyarrow.lib.DictionaryArray object at 0x7f2f399f2ff0>
-- dictionary:
[
"foo",
"bar"
]
-- indices:
[]
In [9]: pc.take(arr[:0], pa.array([], pa.int64()))
Out[9]:
<pyarrow.lib.DictionaryArray object at 0x7f2f3b048040>
-- dictionary:
[
"foo",
"bar"
]
-- indices:
[]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "LarryHu0217 (via GitHub)" <gi...@apache.org>.
LarryHu0217 commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-5234264899
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953860884
I agree that the original dictionary should probably be preserved for consistency, if it's not too costly to do so.
cc @felipecrv @jorisvandenbossche
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Casting an empty dictionary array loses category information [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #40128:
URL: https://github.com/apache/arrow/issues/40128#issuecomment-1953891027
@stinodego Do you think you would be able to work on a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org