You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/01/27 01:51:18 UTC
[PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Yicong-Huang opened a new pull request, #53994:
URL: https://github.com/apache/spark/pull/53994
### What changes were proposed in this pull request?
Fix PyArrow version compatibility for struct cast tests in `test_pyarrow_array_cast.py`.
Some struct cast behaviors differ across PyArrow versions:
| Feature | PyArrow < 19 | PyArrow 19-20 | PyArrow >= 21 |
|---------|--------------|---------------|---------------|
| struct cast: field name mismatch (missing fields → null) | ArrowTypeError | supported | supported |
| struct cast: target has more fields (extra fields → null) | ArrowTypeError | supported | supported |
| struct cast: field reorder (same fields, different order) | ArrowTypeError | ArrowTypeError | supported |
This PR:
1. Adds version compatibility documentation to the test file header
2. Introduces `pyarrow_19_or_greater` and `pyarrow_21_or_greater` version flags
3. Updates `test_struct_casts` to expect `ArrowTypeError` for unsupported operations on older PyArrow versions
### Why are the changes needed?
The existing tests assumed all struct cast operations were supported in all PyArrow versions, causing test failures on PyArrow 18-20.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested with PyArrow 18, 19, 20, 21, and 22 - all versions pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53994:
URL: https://github.com/apache/spark/pull/53994#issuecomment-3802648543
## JIRA Issue Information
=== Sub-task SPARK-54942 ===
Summary: Add tests for pa.Array.cast with default arguments
Assignee: Yicong Huang
Status: Resolved
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #53994:
URL: https://github.com/apache/spark/pull/53994#issuecomment-3802648725
cc @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #53994: [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility
URL: https://github.com/apache/spark/pull/53994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #53994:
URL: https://github.com/apache/spark/pull/53994#discussion_r2730040527
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_cast.py:
##########
@@ -38,13 +38,31 @@
- **List variants**: list, large_list, fixed_size_list
- **Map**: map<key, value>
- **Struct**: struct<fields...>
+
+## PyArrow Version Compatibility
Review Comment:
Thank you. I'm thinking of the end of this page because `PyArrow >=18.0.0` requirement is documented there.
- https://github.com/apache/spark/blob/master/python/docs/source/getting_started/install.rst
- https://spark.apache.org/docs/4.1.1/api/python/getting_started/install.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53994:
URL: https://github.com/apache/spark/pull/53994#discussion_r2730029662
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_cast.py:
##########
@@ -38,13 +38,31 @@
- **List variants**: list, large_list, fixed_size_list
- **Map**: map<key, value>
- **Struct**: struct<fields...>
+
+## PyArrow Version Compatibility
Review Comment:
We are in the process to add those monitoring tests to detect changes like this. I'd be happy to put this information in the doc. which file/page would be a good place to document it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #53994:
URL: https://github.com/apache/spark/pull/53994#discussion_r2730025589
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_cast.py:
##########
@@ -38,13 +38,31 @@
- **List variants**: list, large_list, fixed_size_list
- **Map**: map<key, value>
- **Struct**: struct<fields...>
+
+## PyArrow Version Compatibility
Review Comment:
Thank you for clarifying this, @Yicong-Huang .
Do you think we can have this in PySpark documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53994:
URL: https://github.com/apache/spark/pull/53994#discussion_r2730047195
##########
python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_cast.py:
##########
@@ -38,13 +38,31 @@
- **List variants**: list, large_list, fixed_size_list
- **Map**: map<key, value>
- **Struct**: struct<fields...>
+
+## PyArrow Version Compatibility
Review Comment:
will update the doc in another PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #53994:
URL: https://github.com/apache/spark/pull/53994#issuecomment-3803130883
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org