You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "spenczar (via GitHub)" <gi...@apache.org> on 2023/06/01 03:47:51 UTC
[GitHub] [arrow] spenczar opened a new pull request, #35864: GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray
spenczar opened a new pull request, #35864:
URL: https://github.com/apache/arrow/pull/35864
### What changes are included in this PR?
#35747 asks for additions to all three ListArray types (ListArray, FixedSizeListArray, and LargeListArray). This PR only adds for FixedSizeListArray. That's just because it's both the easiest to add, and the most obviously useful scenario.
This implementation is just in Python and follows the general pattern of FixedShapeTensorArrays.
An alternative implementation would be to do this in [python/pyarrow/src/arrow/python/numpy_convert.cc](https://github.com/apache/arrow/blob/cbc17a98dfd9fb660523f61ecd361b157d278372/python/pyarrow/src/arrow/python/numpy_convert.cc). We could detect 2D Numpy arrays and convert them to FixedShapeTensorArrays. That would be a much more significant change; I think it would change the behavior of other calls like `pyarrow.array()`. It would also allow for much more careful management of copies and memory; it should be possible to implement both to_ and from_ with zero copy for primitive numeric types. However, it would be a lot more complex, and this is my first Arrow contribution, so I figured I'd start a bit smaller.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, and I think I added sufficient documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #35864: GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #35864:
URL: https://github.com/apache/arrow/pull/35864#issuecomment-1596733369
Thank you @spenczar for making a contribution!
And sorry it took a while to get a review.
The implementation in Python instead of C++ makes sense. It might also make sense to think about the value of having this case (numpy arrays of 1 vs 2 dimensions, how one could still use 2-dim arrays in pyarrow avoiding an error) added to the documentation without adding two new functions to the code. I think that could be enough for a user needing to have an option for 2-dim numpy arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #35864:
URL: https://github.com/apache/arrow/pull/35864#issuecomment-2100475585
Hey @spenczar, would you consider:
`FixedSizeListArray`:
- "casting" to `pa.FixedShapeTensorArray` and then using `to_numpy`
- `pa.FixedShapeTensorArray.from_numpy_ndarray` and then getting the fixed sized list type storage?
`ListArray/LargeListArray`:
- "casting" to `pa.VariableShapeTensor` and then using `to_numpy` (once https://github.com/apache/arrow/pull/40354 is merged)
- `pa.VariableShapeTensorArray.from_numpy_ndarray` anf then geting the list type storage?
This could work either currently as a workaround or as machinery for this PR.
In the latter case there's a minor caveat - `FixedShapeTensorArray` and `VariableShapeTensorArray` are available only if json is enabled ad compile time (`-DARROW_JSON=ON`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #35864: GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35864:
URL: https://github.com/apache/arrow/pull/35864#issuecomment-1571293655
:warning: GitHub issue #35747 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray [arrow]
Posted by "spenczar (via GitHub)" <gi...@apache.org>.
spenczar commented on PR #35864:
URL: https://github.com/apache/arrow/pull/35864#issuecomment-3324664482
Cleaning up old un-merged PRs in my queue. I still think this was a good idea but it appears to have no movement; I understand it is blocked by #40354.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #35864:
URL: https://github.com/apache/arrow/pull/35864#issuecomment-3325136091
I hope we cycle back to this once #40354 is merged. And sorry for the wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray [arrow]
Posted by "spenczar (via GitHub)" <gi...@apache.org>.
spenczar closed pull request #35864: GH-35747: Add FixedSizeListArray.to_numpy_ndarray and FixedSizeListArray.from_numpy_ndarray
URL: https://github.com/apache/arrow/pull/35864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org