You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "itholic (via GitHub)" <gi...@apache.org> on 2023/12/02 12:34:44 UTC
[PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
itholic opened a new pull request, #44123:
URL: https://github.com/apache/spark/pull/44123
### What changes were proposed in this pull request?
This PR proposes to introduce `PySparkImportError` for error framework to replace `IndexError` in pyspark.sql.*.
### Why are the changes needed?
For better error handling.
### Does this PR introduce _any_ user-facing change?
No API changes, but it's improve the user-facing error messages.
### How was this patch tested?
The existing CI should pass
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #44123:
URL: https://github.com/apache/spark/pull/44123#discussion_r1414742528
##########
python/pyspark/errors/error_classes.py:
##########
@@ -667,6 +667,11 @@
"Only a single trigger is allowed."
]
},
+ "PACKAGE_NOT_INSTALLED" : {
+ "message" : [
+ "<package_name> >= <minimum_version> must be installed; however, it was not found."
Review Comment:
Oh,,, seems like the special character `>` somehow breaks the build while parsing the error message format. Let me made a followup PR to fix it. Thanks for addressing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44123:
URL: https://github.com/apache/spark/pull/44123#discussion_r1414755240
##########
python/pyspark/errors/error_classes.py:
##########
@@ -667,6 +667,11 @@
"Only a single trigger is allowed."
]
},
+ "PACKAGE_NOT_INSTALLED" : {
+ "message" : [
+ "<package_name> >= <minimum_version> must be installed; however, it was not found."
Review Comment:
Oh..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #44123: [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework
URL: https://github.com/apache/spark/pull/44123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #44123:
URL: https://github.com/apache/spark/pull/44123#discussion_r1414734799
##########
python/pyspark/errors/error_classes.py:
##########
@@ -667,6 +667,11 @@
"Only a single trigger is allowed."
]
},
+ "PACKAGE_NOT_INSTALLED" : {
+ "message" : [
+ "<package_name> >= <minimum_version> must be installed; however, it was not found."
Review Comment:
@itholic, the code here fails:
```python
>>> from pyspark.sql.pandas.utils import require_minimum_pyarrow_version
>>> require_minimum_pyarrow_version()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/sql/pandas/utils.py", line 68, in require_minimum_pyarrow_version
raise PySparkImportError(
File "/.../spark/python/pyspark/errors/exceptions/base.py", line 46, in __init__
self._message = self._error_reader.get_error_message(
File "/.../spark/python/pyspark/errors/utils.py", line 39, in get_error_message
assert set(message_parameters_from_template) == set(message_parameters), (
AssertionError: Undefined error message parameter for error class: PACKAGE_NOT_INSTALLED. Parameters: {'package_name:': 'PyArrow', 'minimum_version': '4.0.0'}
```
This breaks scheduled Python build https://github.com/apache/spark/actions/runs/7088462269/job/19291103742. I an reverting this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #44123:
URL: https://github.com/apache/spark/pull/44123#issuecomment-1837309298
> Since I'm not sure, I simply remove the following part from the PR description.
Yeah, that was my mistake. Thanks for the correction!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #44123:
URL: https://github.com/apache/spark/pull/44123#discussion_r1414742528
##########
python/pyspark/errors/error_classes.py:
##########
@@ -667,6 +667,11 @@
"Only a single trigger is allowed."
]
},
+ "PACKAGE_NOT_INSTALLED" : {
+ "message" : [
+ "<package_name> >= <minimum_version> must be installed; however, it was not found."
Review Comment:
Oh,,, seems like the special character `>` somehow breaks the build while parsing the error message format. Let me made a followup PR to fix it. Thanks for investigating this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44123:
URL: https://github.com/apache/spark/pull/44123#issuecomment-1837255140
Since I'm not sure, I simply remove this part, `to replace `IndexError` in pyspark.sql.*.` from the PR description.
Merged to master~ Thank you, @itholic .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44123:
URL: https://github.com/apache/spark/pull/44123#discussion_r1414755349
##########
python/pyspark/errors/error_classes.py:
##########
@@ -667,6 +667,11 @@
"Only a single trigger is allowed."
]
},
+ "PACKAGE_NOT_INSTALLED" : {
+ "message" : [
+ "<package_name> >= <minimum_version> must be installed; however, it was not found."
Review Comment:
Thank you for reverting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org