You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/16 11:45:02 UTC
[GitHub] [spark] itholic opened a new pull request, #38673: [SPARK-41149][PYTHON] Fix `SparkSession.builder.config` to support bool
itholic opened a new pull request, #38673:
URL: https://github.com/apache/spark/pull/38673
### What changes were proposed in this pull request?
This PR proposes to support `bool` type for `SparkSession.builder.config` when building the new `SparkSession`.
### Why are the changes needed?
Currently, Python `bool` type `True` or `False` is not working correctly, since JVM side only treat the String type of "true" or "false" as a valid option value.
So, this PR basically fix the current behavior as below:
**Before**
```python
>>> s1 = spark.builder.config("spark.sql.pyspark.jvmStacktrace.enabled", True).getOrCreate()
>>> s1.conf.get("spark.sql.pyspark.jvmStacktrace.enabled")
'True' # invalid value, no effect for options.
```
**After**
```python
>>> s1 = spark.builder.config("spark.sql.pyspark.jvmStacktrace.enabled", True).getOrCreate()
>>> s1.conf.get("spark.sql.pyspark.jvmStacktrace.enabled")
'true' # the lower case "true" only valid for options.
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unittest added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38673: [SPARK-41149][PYTHON] Fix `SparkSession.builder.config` to support bool
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38673:
URL: https://github.com/apache/spark/pull/38673#issuecomment-1318128005
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38673: [SPARK-41149][PYTHON] Fix `SparkSession.builder.config` to support bool
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38673:
URL: https://github.com/apache/spark/pull/38673#discussion_r1024678857
##########
python/pyspark/sql/session.py:
##########
@@ -256,8 +256,12 @@ def config(
self._options[k] = v
elif map is not None:
for k, v in map.items(): # type: ignore[assignment]
+ if isinstance(v, bool):
+ v = "true" if v is True else "false"
self._options[k] = str(v)
else:
+ if isinstance(value, bool):
+ value = "true" if value is True else "false"
Review Comment:
Can you use `from pyspark.sql.utils import to_str`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #38673: [SPARK-41149][PYTHON] Fix `SparkSession.builder.config` to support bool
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #38673: [SPARK-41149][PYTHON] Fix `SparkSession.builder.config` to support bool
URL: https://github.com/apache/spark/pull/38673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org