You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2025/05/13 09:27:30 UTC
[PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
rok opened a new pull request, #46423:
URL: https://github.com/apache/arrow/pull/46423
### Rationale for this change
As per [SciPy's migration guide](https://scipy.github.io/devdocs/reference/sparse.migration_to_sparray.html) it would be good to change the constructors we use when creating scipy's sparse object. Further we should allow for creating sparse arrow objects from `scipy.sparray` (currently only `scipy.spmatrix)` objects and test for it.
### What changes are included in this PR?
This adjusts `pa.SparseCXXMatrix` constructors and `pa.SparseCXXMatrix.to_scipy` to achieve goals listed above and adds tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
`pyarrow.coo_matrix.to_scipy` and `pyarrow.csr_matrix.to_scipy` would now return different types (sparray instead of spmatrix) which would probably be a breaking change (behaviors of the types would be similar).
Additionally `pa.SparseCXXMatrix.from_scipy` would now take `scipy.sparray` and `scipy.spmatrix` which would not be a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok merged PR #46423:
URL: https://github.com/apache/arrow/pull/46423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-3032845181
Sorry this took a while, my local environment was broken for some reason.
With this PR pyarrow scipy tests will be skipped on scipy<1.8.0.
`to_scipy` methods will return `xxx_array` instead of `xxx_matrix` so I suppose this will be a breaking change, but I'm not sure how destructive it'll be downstream.
If there's now objections I'll merge tomorrow morning or when CI is green whichever happens last.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #46423:
URL: https://github.com/apache/arrow/pull/46423#discussion_r2183311621
##########
python/pyarrow/tensor.pxi:
##########
@@ -732,19 +733,19 @@ shape: {self.shape}"""
@staticmethod
def from_scipy(obj, dim_names=None):
"""
- Convert scipy.sparse.csr_matrix to arrow::SparseCSRMatrix.
+ Convert scipy.sparse.csr_array or scipy.sparse.csr_matrix to arrow::SparseCSRMatrix.
Parameters
----------
- obj : scipy.sparse.csr_matrix
+ obj : scipy.sparse.csr_array or scipy.sparse.csr_matrix
The scipy matrix that should be converted.
dim_names : list, optional
Names of the dimensions.
"""
import scipy.sparse
- if not isinstance(obj, scipy.sparse.csr_matrix):
+ if not (isinstance(obj, scipy.sparse.csr_array) or isinstance(obj, scipy.sparse.csr_matrix)):
Review Comment:
Done
##########
python/pyarrow/tensor.pxi:
##########
@@ -973,19 +974,19 @@ shape: {self.shape}"""
@staticmethod
def from_scipy(obj, dim_names=None):
"""
- Convert scipy.sparse.csc_matrix to arrow::SparseCSCMatrix
+ Convert scipy.sparse.csc_array or scipy.sparse.csc_matrix to arrow::SparseCSCMatrix
Parameters
----------
- obj : scipy.sparse.csc_matrix
+ obj : scipy.sparse.csc_array or scipy.sparse.csc_matrix
The scipy matrix that should be converted.
dim_names : list, optional
Names of the dimensions.
"""
import scipy.sparse
- if not isinstance(obj, scipy.sparse.csc_matrix):
+ if not (isinstance(obj, scipy.sparse.csc_array) or isinstance(obj, scipy.sparse.csc_matrix)):
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-3031204064
> I'm yet to test this locally with different versions of scipy (will do today hopefully). I expect this to break with for pre-[1.8 scipy versions](https://docs.scipy.org/doc/scipy/release/1.8.0-notes.html), released Feb 6, 2022, but I imagine that's ok?
It's up to you to decide, I think, but intuitively that sounds ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-3029818376
I'm yet to test this locally with different versions of scipy (will do today hopefully). I expect this to break with for pre-[1.8 scipy versions](https://docs.scipy.org/doc/scipy/release/1.8.0-notes.html), released Feb 6, 2022, but I imagine that's ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #46423:
URL: https://github.com/apache/arrow/pull/46423#discussion_r2183281348
##########
python/pyarrow/tensor.pxi:
##########
@@ -421,19 +421,19 @@ shape: {self.shape}"""
@staticmethod
def from_scipy(obj, dim_names=None):
"""
- Convert scipy.sparse.coo_matrix to arrow::SparseCOOTensor
+ Convert scipy.sparse.coo_array or scipy.sparse.coo_matrix to arrow::SparseCOOTensor
Parameters
----------
- obj : scipy.sparse.csr_matrix
- The scipy matrix that should be converted.
+ obj : scipy.sparse.coo_array or scipy.sparse.coo_matrix
+ The scipy array or matrix that should be converted.
dim_names : list, optional
Names of the dimensions.
"""
import scipy.sparse
- if not isinstance(obj, scipy.sparse.coo_matrix):
+ if not (isinstance(obj, scipy.sparse.coo_array) or isinstance(obj, scipy.sparse.coo_matrix)):
Review Comment:
More succinctly, you can use `isinstance` with a tuple of classes.
```suggestion
if not isinstance(obj, (scipy.sparse.coo_array, scipy.sparse.coo_matrix)):
```
##########
python/pyarrow/tensor.pxi:
##########
@@ -732,19 +733,19 @@ shape: {self.shape}"""
@staticmethod
def from_scipy(obj, dim_names=None):
"""
- Convert scipy.sparse.csr_matrix to arrow::SparseCSRMatrix.
+ Convert scipy.sparse.csr_array or scipy.sparse.csr_matrix to arrow::SparseCSRMatrix.
Parameters
----------
- obj : scipy.sparse.csr_matrix
+ obj : scipy.sparse.csr_array or scipy.sparse.csr_matrix
The scipy matrix that should be converted.
dim_names : list, optional
Names of the dimensions.
"""
import scipy.sparse
- if not isinstance(obj, scipy.sparse.csr_matrix):
+ if not (isinstance(obj, scipy.sparse.csr_array) or isinstance(obj, scipy.sparse.csr_matrix)):
Review Comment:
Same
##########
python/pyarrow/tensor.pxi:
##########
@@ -973,19 +974,19 @@ shape: {self.shape}"""
@staticmethod
def from_scipy(obj, dim_names=None):
"""
- Convert scipy.sparse.csc_matrix to arrow::SparseCSCMatrix
+ Convert scipy.sparse.csc_array or scipy.sparse.csc_matrix to arrow::SparseCSCMatrix
Parameters
----------
- obj : scipy.sparse.csc_matrix
+ obj : scipy.sparse.csc_array or scipy.sparse.csc_matrix
The scipy matrix that should be converted.
dim_names : list, optional
Names of the dimensions.
"""
import scipy.sparse
- if not isinstance(obj, scipy.sparse.csc_matrix):
+ if not (isinstance(obj, scipy.sparse.csc_array) or isinstance(obj, scipy.sparse.csc_matrix)):
Review Comment:
Same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-2879141940
> Another thing to consider is version compatibility. It looks like the sparse arrays were added in [scipy 1.8.0](https://docs.scipy.org/doc/scipy/release/1.8.0-notes.html) so that would be a new lower bound. Again I'm not sure if we have a way to document this
That's a good point. I suppose we should check for scipy version when returning from `to_scipy` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-3033115681
@pitrou Thanks for the review, I went ahead and merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-3033959078
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit ff0d044b0177418653d19cfb2df4eb3eb132cb92.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/45331401547) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-3024517201
@rok Do you want to update this before it's too late for 21.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45229: [Python] Migrate from scipy.spmatrix to scipy.sparray [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46423:
URL: https://github.com/apache/arrow/pull/46423#issuecomment-2877161909
@TheNeuralBit does this look ok to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org