You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2024/03/19 09:30:42 UTC
[PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
AlenkaF opened a new pull request, #40661:
URL: https://github.com/apache/arrow/pull/40661
### Rationale for this change
We want to enable the conversion of Run-End Encoded arrays to numpy and pandas.
### What changes are included in this PR?
In case of RunEndEncodedArray we first decode the array and then convert the decoded array to numpy or pandas.
### Are these changes tested?
Yes, in Python.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40661:
URL: https://github.com/apache/arrow/pull/40661#discussion_r1530337896
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2320,6 +2329,17 @@ class ConsolidatedBlockCreator : public PandasBlockCreator {
if (arrays_[column_index]->type()->id() == Type::EXTENSION) {
arrays_[column_index] = GetStorageChunkedArray(arrays_[column_index]);
}
+ // In case of a RunEndEncodedArray default to the storage type
+ else if (arrays_[column_index]->type()->id() == Type::RUN_END_ENCODED) {
+ ARROW_ASSIGN_OR_RAISE(Datum decoded,
+ compute::RunEndDecode(arrays_[column_index]));
Review Comment:
> Or the problem is that in this case it can fail and we need to handle that?
Yes, exactly. I thought it was easier to just move it out. If `GetDecodedChunkedArray` stays minimal I thought we might not even use the helper function at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #40661:
URL: https://github.com/apache/arrow/pull/40661#issuecomment-2007050010
For the memory issue, you might need something similar as here:
https://github.com/apache/arrow/blob/a364e4a16d596f7fcccd76ad8fc27092c59e5e74/python/pyarrow/src/arrow/python/arrow_to_pandas.cc#L2527-L2530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #40661:
URL: https://github.com/apache/arrow/pull/40661#issuecomment-2006507006
Opened a draft PR in case there are any ideas what is missing in the C++ code. The conversion currently results in a numpy array or pandas series with correct length but only one value, which is not correct:
```python
In [1]: import pyarrow as pa
...: import pyarrow.compute as pc
...:
...: arr = pc.run_end_encode([1, 1, 2, 3, 3, 3, 6])
...: arr.to_numpy()
Out[1]:
array([6510615555426900570, 6510615555426900570, 6510615555426900570,
6510615555426900570, 6510615555426900570, 6510615555426900570,
6510615555426900570])
In [2]: import pyarrow as pa
...: import pyarrow.compute as pc
...:
...: arr = pc.run_end_encode([1, 1, 2, 3, 3, 3, 6])
...: arr.to_pandas()
Out[2]:
0 6510615555426900570
1 6510615555426900570
2 6510615555426900570
3 6510615555426900570
4 6510615555426900570
5 6510615555426900570
6 6510615555426900570
dtype: int64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40661:
URL: https://github.com/apache/arrow/pull/40661#discussion_r1530022681
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2320,6 +2329,16 @@ class ConsolidatedBlockCreator : public PandasBlockCreator {
if (arrays_[column_index]->type()->id() == Type::EXTENSION) {
arrays_[column_index] = GetStorageChunkedArray(arrays_[column_index]);
}
+ // In case of a RunEndEncodedArray default to the storage type
+ else if (arrays_[column_index]->type()->id() == Type::RUN_END_ENCODED) {
+ ARROW_ASSIGN_OR_RAISE(Datum decoded,
+ compute::RunEndDecode(arrays_[column_index]));
+ // ARROW_ASSIGN_OR_RAISE(
+ // Datum decoded, arrays_[column_index]->num_chunks() > 1
+ // ? compute::RunEndDecode(arrays_[column_index])
+ // : compute::RunEndDecode(arrays_[column_index]->chunk(0)));
Review Comment:
Keeping all the cases where `ChunkedArray` might have one chunk commented out in case it will be needed. I do not think it will be, but saw it being used in https://github.com/apache/arrow/blob/6c06b37434f865e52c69951ad9de40b36895be0d/cpp/src/arrow/compute/kernels/vector_run_end_encode_test.cc#L209-L212 and https://github.com/apache/arrow/blob/6c06b37434f865e52c69951ad9de40b36895be0d/cpp/src/arrow/compute/kernels/vector_run_end_encode_test.cc#L170-L176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40661:
URL: https://github.com/apache/arrow/pull/40661#discussion_r1530258570
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2320,6 +2329,17 @@ class ConsolidatedBlockCreator : public PandasBlockCreator {
if (arrays_[column_index]->type()->id() == Type::EXTENSION) {
arrays_[column_index] = GetStorageChunkedArray(arrays_[column_index]);
}
+ // In case of a RunEndEncodedArray default to the storage type
+ else if (arrays_[column_index]->type()->id() == Type::RUN_END_ENCODED) {
+ ARROW_ASSIGN_OR_RAISE(Datum decoded,
+ compute::RunEndDecode(arrays_[column_index]));
Review Comment:
Can you move this part into the `GetDecodedChunkedArray` helper? (so it is fully equivalent like the `GetStorageChunkedArray` one? Or the problem is that in this case it can fail and we need to handle that?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #40661:
URL: https://github.com/apache/arrow/pull/40661#issuecomment-2012902043
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #40661:
URL: https://github.com/apache/arrow/pull/40661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40661:
URL: https://github.com/apache/arrow/pull/40661#discussion_r1531617792
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2320,6 +2328,11 @@ class ConsolidatedBlockCreator : public PandasBlockCreator {
if (arrays_[column_index]->type()->id() == Type::EXTENSION) {
arrays_[column_index] = GetStorageChunkedArray(arrays_[column_index]);
}
+ // In case of a RunEndEncodedArray default to the storage type
Review Comment:
```suggestion
// In case of a RunEndEncodedArray default to the values type
```
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2554,6 +2567,14 @@ Status ConvertChunkedArrayToPandas(const PandasOptions& options,
if (arr->type()->id() == Type::EXTENSION) {
arr = GetStorageChunkedArray(arr);
}
+ // In case of a RunEndEncodedArray decode the array
+ else if (arr->type()->id() == Type::RUN_END_ENCODED) {
+ ARROW_ASSIGN_OR_RAISE(arr, GetDecodedChunkedArray(arr));
Review Comment:
We probably should check here for `options.zero_copy_only` and raise an error if set to true (like above is done in case of `strings_to_categorical`)
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3580,6 +3580,26 @@ def test_run_end_encoded_from_buffers():
1, offset, children)
+def _generate_ree_array():
+ run_ends = [1, 3, 6]
+ values = [1, 2, 3]
+ ree_type = pa.run_end_encoded(pa.int32(), pa.int64())
+ return pa.RunEndEncodedArray.from_arrays(run_ends, values,
Review Comment:
Since we are not testing the constructor here (and now I merged your other PR), I think doing something simpler as `pa.array([1, 2, 2, 3, 3, 3], pa.run_end_encoded(pa.int32(), pa.int64()))` is also fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #40661:
URL: https://github.com/apache/arrow/pull/40661#discussion_r1531596575
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2320,6 +2329,17 @@ class ConsolidatedBlockCreator : public PandasBlockCreator {
if (arrays_[column_index]->type()->id() == Type::EXTENSION) {
arrays_[column_index] = GetStorageChunkedArray(arrays_[column_index]);
}
+ // In case of a RunEndEncodedArray default to the storage type
+ else if (arrays_[column_index]->type()->id() == Type::RUN_END_ENCODED) {
+ ARROW_ASSIGN_OR_RAISE(Datum decoded,
+ compute::RunEndDecode(arrays_[column_index]));
Review Comment:
Done: https://github.com/apache/arrow/pull/40661/commits/6308551254279f4387da4c0ceb6bf8b3edc6de90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40661:
URL: https://github.com/apache/arrow/pull/40661#discussion_r1530447135
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2320,6 +2329,17 @@ class ConsolidatedBlockCreator : public PandasBlockCreator {
if (arrays_[column_index]->type()->id() == Type::EXTENSION) {
arrays_[column_index] = GetStorageChunkedArray(arrays_[column_index]);
}
+ // In case of a RunEndEncodedArray default to the storage type
+ else if (arrays_[column_index]->type()->id() == Type::RUN_END_ENCODED) {
+ ARROW_ASSIGN_OR_RAISE(Datum decoded,
+ compute::RunEndDecode(arrays_[column_index]));
Review Comment:
I think you should be able to write it such that `GetDecodedChunkedArray` returns a Result, and then still use that here. For example, it should be possible to do something like `ARROW_ASSIGN_OR_RAISE(arrays_[column_index], GetDecodedChunkedArray(arrays_[column_index]))`, I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-40659: [Python][C++] Support conversion of pyarrow.RunEndEncodedArray to numpy/pandas [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40661:
URL: https://github.com/apache/arrow/pull/40661#issuecomment-2013738872
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit dada4e1aad62bdfe9c37112a005bd1b84c5cbd8f.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/22954083050) has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org