You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/14 07:07:21 UTC
[GitHub] [spark] yoda-mon opened a new pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
yoda-mon opened a new pull request #33992:
URL: https://github.com/apache/spark/pull/33992
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
octet_length: caliculate the byte length of strings
bit_length: caliculate the bit length of strings
Those two string related functions are only implemented on SparkSQL, not on Scala, Python and R.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Those functions would be useful for multi-bytes character users, who mainly working with Scala, Python or R.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. Users can call octet_length/bit_length APIs on Scala(Dataframe), Python, and R.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919016123
**[Test build #143253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143253/testReport)** for PR 33992 at commit [`3a07337`](https://github.com/apache/spark/commit/3a0733747c3db00ae4e5fcfd3432be3bf8ae9325).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919619804
**[Test build #143288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143288/testReport)** for PR 33992 at commit [`afa1700`](https://github.com/apache/spark/commit/afa1700ee3f4e3a0ba0f0a685c091e45f1631a09).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918947140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47752/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708052662
##########
File path: python/pyspark/sql/tests/test_functions.py
##########
@@ -197,6 +197,18 @@ def test_string_functions(self):
df.select(getattr(functions, name)("name")).first()[0],
df.select(getattr(functions, name)(col("name"))).first()[0])
+ def test_octet_length_function(self):
+ from pyspark.sql.functions import octet_length
+ df = self.spark.createDataFrame([('cat',), ('\U0001F408',)], ['cat'])
+ actual = df.select(octet_length('cat')).collect()
+ self.assertEqual([Row(3), Row(4)], actual)
+
+ def test_bit_length_function(self):
+ from pyspark.sql.functions import bit_length
Review comment:
I know this is sort of consistent with other tests but in fact it's totally fine to import it on the top. Since this is a new feature that's not going to be backported, let's move this import to the top.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919654177
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919651180
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918947101
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47752/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919745703
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143290/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919161811
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47763/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919082747
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918872015
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918899691
**[Test build #143249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143249/testReport)** for PR 33992 at commit [`c0ffdce`](https://github.com/apache/spark/commit/c0ffdce41df280ab1bfa6b3b43dc375352ea3adf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708195901
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
##########
@@ -486,6 +486,58 @@ class StringFunctionsSuite extends QueryTest with SharedSparkSession {
)
}
+ test("SPARK-36751: add octet/bit length api for scala") {
+ // scalastyle:off
+ // non ascii characters are not allowed in the code, so we disable the scalastyle here.
+ val df = Seq(("123", Array[Byte](1, 2, 3, 4), 123, 2.0f, 3.015, "\ud83d\udc08"))
+ .toDF("a", "b", "c", "d", "e", "f")
+ // string and binary input
+ checkAnswer(
+ df.select(octet_length($"a"), octet_length($"b")),
+ Row(3, 4))
+ // string and binary input
+ checkAnswer(
+ df.selectExpr("octet_length(a)", "octet_length(b)"),
+ Row(3, 4))
+ // integer, float and double input
+ checkAnswer(
+ df.selectExpr("octet_length(c)", "octet_length(d)", "octet_length(e)"),
+ Row(3, 3, 5)
+ )
+ // multi-byte character input
+ checkAnswer(
+ df.selectExpr("octet_length(f)"),
+ Row(4)
+ )
+ // scalastyle:on
+ }
+
+ test("SPARK-36751: add octet/bit length api for scala") {
Review comment:
Oh, you need to give a different name for each test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708347128
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,52 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ Calculates the byte length for the specified string column.
+
+ .. versionadded:: 3.3.0
Review comment:
```suggestion
.. versionadded:: 3.3.0
```
##########
File path: R/pkg/R/functions.R
##########
@@ -646,6 +646,19 @@ setMethod("bin",
column(jc)
})
+#' @details
+#' \code{bit_length}: Calculates the bit length for the specified string column.
Review comment:
```suggestion
#' \code{bit_length}: Calculates the bit length for the specified string column.
```
##########
File path: R/pkg/R/functions.R
##########
@@ -1569,6 +1582,19 @@ setMethod("negate",
column(jc)
})
+#' @details
+#' \code{octet_length}: Calculates the byte length for the specified string column.
Review comment:
```suggestion
#' \code{octet_length}: Calculates the byte length for the specified string column.
```
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,52 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ Calculates the byte length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
Review comment:
```suggestion
Source column or strings
```
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,52 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ Calculates the byte length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Byte length of the col
Review comment:
```suggestion
Byte length of the col
```
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,52 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ Calculates the byte length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Byte length of the col
+ Examples
+ -------
+ >>> from pyspark.sql.functions import octet_length
+ >>> spark.createDataFrame([('cat',), ( '\U0001F408',)], ['cat']) \
+ .select(octet_length('cat')).collect()
+ [Row(octet_length(cat)=3), Row(octet_length(cat)=4)]
+ """
+ return _invoke_function_over_column("octet_length", col)
+
+
+def bit_length(col):
+ """
+ Calculates the bit length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Bit length of the col
Review comment:
```suggestion
Bit length of the col
```
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,52 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ Calculates the byte length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Byte length of the col
+ Examples
+ -------
+ >>> from pyspark.sql.functions import octet_length
+ >>> spark.createDataFrame([('cat',), ( '\U0001F408',)], ['cat']) \
+ .select(octet_length('cat')).collect()
+ [Row(octet_length(cat)=3), Row(octet_length(cat)=4)]
+ """
+ return _invoke_function_over_column("octet_length", col)
+
+
+def bit_length(col):
+ """
+ Calculates the bit length for the specified string column.
+
+ .. versionadded:: 3.3.0
Review comment:
```suggestion
.. versionadded:: 3.3.0
```
##########
File path: python/pyspark/sql/tests/test_functions.py
##########
@@ -197,6 +197,18 @@ def test_string_functions(self):
df.select(getattr(functions, name)("name")).first()[0],
df.select(getattr(functions, name)(col("name"))).first()[0])
+ def test_octet_length_function(self):
+ # SPARK-36751: add octet/bit length api for python
Review comment:
```suggestion
# SPARK-36751: add octet length api for python
```
##########
File path: python/pyspark/sql/tests/test_functions.py
##########
@@ -197,6 +197,18 @@ def test_string_functions(self):
df.select(getattr(functions, name)("name")).first()[0],
df.select(getattr(functions, name)(col("name"))).first()[0])
+ def test_octet_length_function(self):
+ # SPARK-36751: add octet/bit length api for python
+ df = self.spark.createDataFrame([('cat',), ('\U0001F408',)], ['cat'])
+ actual = df.select(octet_length('cat')).collect()
+ self.assertEqual([Row(3), Row(4)], actual)
+
+ def test_bit_length_function(self):
+ # SPARK-36751: add octet/bit length api for python
Review comment:
```suggestion
# SPARK-36751: add bit length api for python
```
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,52 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ Calculates the byte length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Byte length of the col
+ Examples
+ -------
+ >>> from pyspark.sql.functions import octet_length
+ >>> spark.createDataFrame([('cat',), ( '\U0001F408',)], ['cat']) \
+ .select(octet_length('cat')).collect()
+ [Row(octet_length(cat)=3), Row(octet_length(cat)=4)]
+ """
+ return _invoke_function_over_column("octet_length", col)
+
+
+def bit_length(col):
+ """
+ Calculates the bit length for the specified string column.
+
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
Review comment:
```suggestion
Source column or strings
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919113495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919065720
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47756/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708051394
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
##########
@@ -486,6 +486,59 @@ class StringFunctionsSuite extends QueryTest with SharedSparkSession {
)
}
+ test("octet-length function") {
Review comment:
I would add a JIRA prefix e.g.) `test("SPARK-36751: octet-length function") {`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yoda-mon commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
yoda-mon commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919081851
@HyukjinKwon @sarutak
Thank you for your comments, I fixed the commented points.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919374714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143260/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919175389
**[Test build #143253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143253/testReport)** for PR 33992 at commit [`3a07337`](https://github.com/apache/spark/commit/3a0733747c3db00ae4e5fcfd3432be3bf8ae9325).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919082747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919176630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-920007710
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919071466
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47756/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919745703
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143290/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919633484
**[Test build #143290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143290/testReport)** for PR 33992 at commit [`41656f4`](https://github.com/apache/spark/commit/41656f43bfa41431035bf9023ee6035f856570b3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708053772
##########
File path: python/pyspark/sql/functions.py
##########
@@ -3098,6 +3098,48 @@ def length(col):
return Column(sc._jvm.functions.length(_to_java_column(col)))
+def octet_length(col):
+ """
+ .. versionadded:: 3.3.0
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ Source column or strings
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Byte length of the col
+ Examples
+ -------
+ >>> from pyspark.sql.functions import octet_length
+ >>> spark.createDataFrame([('cat',), ( '\U0001F408',)], ['cat']) \
+ .select(octet_length('cat')).collect()
+ [Row(octet_length(cat)=3), Row(octet_length(cat)=4)]
+ """
+ return _invoke_function_over_column("octet_length", col)
+
+
+def bit_length(col):
+ """
+ .. versionadded:: 3.3.0
+ Parameters
Review comment:
Shall we add a short description and add newline before after `.. versionadded:: 3.3.0`? e.g.)
```
"""
Bit length blah blah
.. versionadded:: 3.3.0
Parameters
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919109818
**[Test build #143249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143249/testReport)** for PR 33992 at commit [`c0ffdce`](https://github.com/apache/spark/commit/c0ffdce41df280ab1bfa6b3b43dc375352ea3adf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919137566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yoda-mon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
yoda-mon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708197825
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
##########
@@ -486,6 +486,58 @@ class StringFunctionsSuite extends QueryTest with SharedSparkSession {
)
}
+ test("SPARK-36751: add octet/bit length api for scala") {
+ // scalastyle:off
+ // non ascii characters are not allowed in the code, so we disable the scalastyle here.
+ val df = Seq(("123", Array[Byte](1, 2, 3, 4), 123, 2.0f, 3.015, "\ud83d\udc08"))
+ .toDF("a", "b", "c", "d", "e", "f")
+ // string and binary input
+ checkAnswer(
+ df.select(octet_length($"a"), octet_length($"b")),
+ Row(3, 4))
+ // string and binary input
+ checkAnswer(
+ df.selectExpr("octet_length(a)", "octet_length(b)"),
+ Row(3, 4))
+ // integer, float and double input
+ checkAnswer(
+ df.selectExpr("octet_length(c)", "octet_length(d)", "octet_length(e)"),
+ Row(3, 3, 5)
+ )
+ // multi-byte character input
+ checkAnswer(
+ df.selectExpr("octet_length(f)"),
+ Row(4)
+ )
+ // scalastyle:on
+ }
+
+ test("SPARK-36751: add octet/bit length api for scala") {
Review comment:
Sorry, I'll fix it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919137566
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47762/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919769668
Merged to `master`. Thanks @yoda-mon and @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak closed pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
sarutak closed pull request #33992:
URL: https://github.com/apache/spark/pull/33992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919164598
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47763/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919718156
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919116352
**[Test build #143260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143260/testReport)** for PR 33992 at commit [`545fbe2`](https://github.com/apache/spark/commit/545fbe2a41cc209171b153111bd58e535ed200ec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919619804
**[Test build #143288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143288/testReport)** for PR 33992 at commit [`afa1700`](https://github.com/apache/spark/commit/afa1700ee3f4e3a0ba0f0a685c091e45f1631a09).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919633484
**[Test build #143290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143290/testReport)** for PR 33992 at commit [`41656f4`](https://github.com/apache/spark/commit/41656f43bfa41431035bf9023ee6035f856570b3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919016123
**[Test build #143253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143253/testReport)** for PR 33992 at commit [`3a07337`](https://github.com/apache/spark/commit/3a0733747c3db00ae4e5fcfd3432be3bf8ae9325).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919091975
**[Test build #143259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143259/testReport)** for PR 33992 at commit [`545fbe2`](https://github.com/apache/spark/commit/545fbe2a41cc209171b153111bd58e535ed200ec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919091975
**[Test build #143259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143259/testReport)** for PR 33992 at commit [`545fbe2`](https://github.com/apache/spark/commit/545fbe2a41cc209171b153111bd58e535ed200ec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918872015
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919176630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919107729
retest this please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919128094
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47762/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918946267
the change looks fine otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919071610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47756/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918899691
**[Test build #143249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143249/testReport)** for PR 33992 at commit [`c0ffdce`](https://github.com/apache/spark/commit/c0ffdce41df280ab1bfa6b3b43dc375352ea3adf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919071610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47756/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919734730
**[Test build #143290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143290/testReport)** for PR 33992 at commit [`41656f4`](https://github.com/apache/spark/commit/41656f43bfa41431035bf9023ee6035f856570b3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919368155
**[Test build #143260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143260/testReport)** for PR 33992 at commit [`545fbe2`](https://github.com/apache/spark/commit/545fbe2a41cc209171b153111bd58e535ed200ec).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r707990771
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
##########
@@ -486,6 +486,59 @@ class StringFunctionsSuite extends QueryTest with SharedSparkSession {
)
}
+ test("octet-length function") {
+ // scalastyle:off
+ // non ascii characters are not allowed in the code, so we disable the scalastyle here.
+ val df = Seq(("123", Array[Byte](1, 2, 3, 4), 123, 2.0f, 3.015, "\ud83d\udc08"))
+ .toDF("a", "b", "c", "d", "e", "f")
+ // string and binary input
+ checkAnswer(
+ df.select(octet_length($"a"), octet_length($"b")),
+ Row(3, 4))
+ // string and binary input
+ checkAnswer(
+ df.selectExpr("octet_length(a)", "octet_length(b)"),
+ Row(3, 4))
+ // integer, float and double input
+ checkAnswer(
+ df.selectExpr("octet_length(c)", "octet_length(d)", "octet_length(e)"),
+ Row(3, 3, 5)
+ )
+ // multi-byte character input
+ checkAnswer(
+ df.selectExpr("octet_length(f)"),
+ Row(4)
+ )
+ // scalastyle:on
+ }
+
+ test("bit-length function") {
+ // scalastyle:off
+ // non ascii characters are not allowed in the code, so we disable the scalastyle here.
+ val df = Seq(("123", Array[Byte](1, 2, 3, 4), 123, 2.0f, 3.015, "\ud83d\udc08"))
+ .toDF("a", "b", "c", "d", "e", "f")
+ // string and binary input
+ checkAnswer(
+ df.select(bit_length($"a"), bit_length($"b")),
+ Row(24, 32))
+ // string and binary input
+ checkAnswer(
+ df.selectExpr("bit_length(a)", "bit_length(b)"),
+ Row(24, 32))
+ // integer, float and double input
+ checkAnswer(
+ df.selectExpr("bit_length(c)", "bit_length(d)", "bit_length(e)"),
+ Row(24, 24, 40)
+ )
+ // multi-byte character input
+ checkAnswer(
+ df.selectExpr("bit_length(f)"),
+ Row(32)
+ )
+ // scalastyle:on
+ }
+
+
Review comment:
Unnecessary blank line here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919137503
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47762/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919116352
**[Test build #143260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143260/testReport)** for PR 33992 at commit [`545fbe2`](https://github.com/apache/spark/commit/545fbe2a41cc209171b153111bd58e535ed200ec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919654275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708354614
##########
File path: python/pyspark/sql/functions.pyi
##########
@@ -174,6 +174,8 @@ def bin(col: ColumnOrName) -> Column: ...
def hex(col: ColumnOrName) -> Column: ...
def unhex(col: ColumnOrName) -> Column: ...
def length(col: ColumnOrName) -> Column: ...
+def octet_length(col: ColumnOrName) -> Column: ...
+def bit_length(col: ColumnOrName) -> Column: ...
Review comment:
Oh we should also add it to PySpark docs - python/docs/source/reference/pyspark.sql.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919640359
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919374714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143260/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918947140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47752/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-918895445
ok to test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919636269
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919101685
**[Test build #143259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143259/testReport)** for PR 33992 at commit [`545fbe2`](https://github.com/apache/spark/commit/545fbe2a41cc209171b153111bd58e535ed200ec).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919654275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919715876
**[Test build #143288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143288/testReport)** for PR 33992 at commit [`afa1700`](https://github.com/apache/spark/commit/afa1700ee3f4e3a0ba0f0a685c091e45f1631a09).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33992:
URL: https://github.com/apache/spark/pull/33992#discussion_r708052189
##########
File path: python/pyspark/sql/tests/test_functions.py
##########
@@ -197,6 +197,18 @@ def test_string_functions(self):
df.select(getattr(functions, name)("name")).first()[0],
df.select(getattr(functions, name)(col("name"))).first()[0])
+ def test_octet_length_function(self):
+ from pyspark.sql.functions import octet_length
+ df = self.spark.createDataFrame([('cat',), ('\U0001F408',)], ['cat'])
+ actual = df.select(octet_length('cat')).collect()
+ self.assertEqual([Row(3), Row(4)], actual)
+
+ def test_bit_length_function(self):
Review comment:
I would also add the comment with JIRA prefix (see "Pull Request" at https://spark.apache.org/contributing.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33992: [SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33992:
URL: https://github.com/apache/spark/pull/33992#issuecomment-919718156
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org