You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/09 09:19:08 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #39001: [SPARK-41402][SQL][CONNECT] Override prettyName of StringDecode
HyukjinKwon opened a new pull request, #39001:
URL: https://github.com/apache/spark/pull/39001
### What changes were proposed in this pull request?
This PR proposes to change `prettyName` of `StringDecode` to `decode` to keep the name as was. It was changed from SPARK-33527.
Note that I don't change `nodeName` here to keep the strings in the plan to differentiate `Decode` from `StringDecode`.
### Why are the changes needed?
This is technically a beahviour change (or regression) from SPARK-33527 although we don't guarantee the compatibility there.
### Does this PR introduce _any_ user-facing change?
Yes, it changes the default output column name from `stringdecode` to `decode`.
### How was this patch tested?
Relates unittests were fixed together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39001: [SPARK-41402][SQL][CONNECT] Override prettyName of StringDecode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39001:
URL: https://github.com/apache/spark/pull/39001#discussion_r1044243371
##########
python/pyspark/sql/connect/functions.py:
##########
@@ -4095,37 +4095,36 @@ def concat_ws(sep: str, *cols: "ColumnOrName") -> Column:
return _invoke_function("concat_ws", lit(sep), *[_to_col(c) for c in cols])
-# TODO: enable with SPARK-41402
-# def decode(col: "ColumnOrName", charset: str) -> Column:
-# """
-# Computes the first argument into a string from a binary using the provided character set
-# (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
-#
-# .. versionadded:: 3.4.0
-#
-# Parameters
-# ----------
-# col : :class:`~pyspark.sql.Column` or str
-# target column to work on.
-# charset : str
-# charset to use to decode to.
-#
-# Returns
-# -------
-# :class:`~pyspark.sql.Column`
-# the column for computed results.
-#
-# Examples
-# --------
-# >>> df = spark.createDataFrame([('abcd',)], ['a'])
-# >>> df.select(decode("a", "UTF-8")).show()
-# +----------------------+
-# |stringdecode(a, UTF-8)|
-# +----------------------+
-# | abcd|
-# +----------------------+
-# """
-# return _invoke_function("decode", _to_col(col), lit(charset))
+def decode(col: "ColumnOrName", charset: str) -> Column:
Review Comment:
I don't mind separating the PR if requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #39001: [SPARK-41402][SQL][CONNECT] Override prettyName of StringDecode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39001:
URL: https://github.com/apache/spark/pull/39001#issuecomment-1344053475
cc @cloud-fan FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #39001: [SPARK-41402][SQL][CONNECT] Override prettyName of StringDecode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39001:
URL: https://github.com/apache/spark/pull/39001#issuecomment-1344404017
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #39001: [SPARK-41402][SQL][CONNECT] Override prettyName of StringDecode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #39001: [SPARK-41402][SQL][CONNECT] Override prettyName of StringDecode
URL: https://github.com/apache/spark/pull/39001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org