You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2023/09/28 07:48:49 UTC
[GitHub] [spark] panbingkun opened a new pull request, #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
panbingkun opened a new pull request, #43168:
URL: https://github.com/apache/spark/pull/43168
### What changes were proposed in this pull request?
The pr aims to fix some UT in `pyspark.sql.functions` when SPARK_ANSI_SQL_MODE=true.
### Why are the changes needed?
Make pyspark test happy.
When Ansi workflow daily GA runs, the following error occurs, eg:
https://github.com/apache/spark/actions/runs/6333367232/job/17202251325
<img width="1011" alt="image" src="https://github.com/apache/spark/assets/15246973/c22fb8c4-8b87-46fd-85f2-51f4c1d8d13d">
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually test:
```
python/run-tests --testnames 'pyspark.sql.functions'
Running PySpark tests. Output is in /Users/panbingkun/Developer/spark/spark-community/python/unit-tests.log
Will test against the following Python executables: ['python3.9']
Will test the following Python tests: ['pyspark.sql.functions']
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.13
Starting test(python3.9): pyspark.sql.functions (temp output: /Users/panbingkun/Developer/spark/spark-community/python/target/c8705d5c-d9f9-4bc5-babf-d3642736c70c/python3.9__pyspark.sql.functions__gcfwu3ik.log)
Finished test(python3.9): pyspark.sql.functions (47s)
Tests passed in 47 seconds
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1349469494
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
>>> df.select(sf.create_map(sf.lit('name'), df['name'],
- ... sf.lit('age'), df['age'])).show(truncate=False)
- +-------------------------+
- |map(name, name, age, age)|
- +-------------------------+
- |{name -> Alice, age -> 2}|
- |{name -> Bob, age -> 5} |
- +-------------------------+
+ ... sf.lit('gender'), df['gender'])).show(truncate=False)
Review Comment:
Will it fail before the modification?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1349440721
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
>>> df.select(sf.create_map(sf.lit('name'), df['name'],
- ... sf.lit('age'), df['age'])).show(truncate=False)
- +-------------------------+
- |map(name, name, age, age)|
- +-------------------------+
- |{name -> Alice, age -> 2}|
- |{name -> Bob, age -> 5} |
- +-------------------------+
+ ... sf.lit('gender'), df['gender'])).show(truncate=False)
Review Comment:
I have manually tested both `export SPARK_ANSI_SQL_MODE=true` and `export SPARK_ANSI_SQL_MODE=false` cases locally
<img width="1420" alt="image" src="https://github.com/apache/spark/assets/15246973/12d89115-9104-45b0-bf1b-55ae72001a91">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1340769111
##########
.github/workflows/build_and_test.yml:
##########
@@ -384,6 +384,7 @@ jobs:
SKIP_PACKAGING: true
METASPACE_SIZE: 1g
BRANCH: ${{ inputs.branch }}
+ SPARK_ANSI_SQL_MODE: true
Review Comment:
After obtaining `approve`, I will set `SPARK_ANSI_SQL_MODE` to false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1339767469
##########
python/pyspark/sql/functions.py:
##########
@@ -11788,50 +11788,54 @@ def array(
Example 1: Basic usage of array function with column names.
>>> from pyspark.sql import functions as sf
- >>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
- >>> df.select(sf.array('name', 'age').alias("arr")).show()
- +----------+
- | arr|
- +----------+
- |[Alice, 2]|
- | [Bob, 5]|
- +----------+
+ >>> df = spark.createDataFrame([("Alice", "doctor"), ("Bob", "engineer")],
+ ... ("name", "occupation"))
+ >>> df.select(sf.array('name', 'occupation').alias("arr")).show()
Review Comment:
made me feel a bit surprised, seems I overlooked the ANSI rules. Thanks @panbingkun
cc @zhengruifeng @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
URL: https://github.com/apache/spark/pull/43168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1339767469
##########
python/pyspark/sql/functions.py:
##########
@@ -11788,50 +11788,54 @@ def array(
Example 1: Basic usage of array function with column names.
>>> from pyspark.sql import functions as sf
- >>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
- >>> df.select(sf.array('name', 'age').alias("arr")).show()
- +----------+
- | arr|
- +----------+
- |[Alice, 2]|
- | [Bob, 5]|
- +----------+
+ >>> df = spark.createDataFrame([("Alice", "doctor"), ("Bob", "engineer")],
+ ... ("name", "occupation"))
+ >>> df.select(sf.array('name', 'occupation').alias("arr")).show()
Review Comment:
made me feel a bit surprised, cc @zhengruifeng @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1341949200
##########
python/pyspark/sql/functions.py:
##########
@@ -11831,50 +11831,54 @@ def array(
Example 1: Basic usage of array function with column names.
>>> from pyspark.sql import functions as sf
- >>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
- >>> df.select(sf.array('name', 'age').alias("arr")).show()
- +----------+
- | arr|
- +----------+
- |[Alice, 2]|
- | [Bob, 5]|
- +----------+
+ >>> df = spark.createDataFrame([("Alice", "doctor"), ("Bob", "engineer")],
+ ... ("name", "occupation"))
+ >>> df.select(sf.array('name', 'occupation').alias("arr")).show()
+ +---------------+
+ | arr|
+ +---------------+
+ |[Alice, doctor]|
+ |[Bob, engineer]|
+ +---------------+
Example 2: Usage of array function with Column objects.
>>> from pyspark.sql import functions as sf
- >>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
- >>> df.select(sf.array(df.name, df.age).alias("arr")).show()
- +----------+
- | arr|
- +----------+
- |[Alice, 2]|
- | [Bob, 5]|
- +----------+
+ >>> df = spark.createDataFrame([("Alice", "doctor"), ("Bob", "engineer")],
Review Comment:
can we have one example that shows type coercion? e.g., float and long should work with ANSI enabled too IIRC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1339682642
##########
.github/workflows/build_and_test.yml:
##########
@@ -384,6 +384,7 @@ jobs:
SKIP_PACKAGING: true
METASPACE_SIZE: 1g
BRANCH: ${{ inputs.branch }}
+ SPARK_ANSI_SQL_MODE: true
Review Comment:
Only for test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1342158979
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
Review Comment:
Why is there a need to modify these two lines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1339767469
##########
python/pyspark/sql/functions.py:
##########
@@ -11788,50 +11788,54 @@ def array(
Example 1: Basic usage of array function with column names.
>>> from pyspark.sql import functions as sf
- >>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
- >>> df.select(sf.array('name', 'age').alias("arr")).show()
- +----------+
- | arr|
- +----------+
- |[Alice, 2]|
- | [Bob, 5]|
- +----------+
+ >>> df = spark.createDataFrame([("Alice", "doctor"), ("Bob", "engineer")],
+ ... ("name", "occupation"))
+ >>> df.select(sf.array('name', 'occupation').alias("arr")).show()
Review Comment:
made me feel a bit surprised, seems I overlooked the ANSI rules. cc @zhengruifeng @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1342159172
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
>>> df.select(sf.create_map(sf.lit('name'), df['name'],
- ... sf.lit('age'), df['age'])).show(truncate=False)
- +-------------------------+
- |map(name, name, age, age)|
- +-------------------------+
- |{name -> Alice, age -> 2}|
- |{name -> Bob, age -> 5} |
- +-------------------------+
+ ... sf.lit('gender'), df['gender'])).show(truncate=False)
Review Comment:
Did this example also fail in ANSI mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43168:
URL: https://github.com/apache/spark/pull/43168#issuecomment-1767915109
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1349476504
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
>>> df.select(sf.create_map(sf.lit('name'), df['name'],
- ... sf.lit('age'), df['age'])).show(truncate=False)
- +-------------------------+
- |map(name, name, age, age)|
- +-------------------------+
- |{name -> Alice, age -> 2}|
- |{name -> Bob, age -> 5} |
- +-------------------------+
+ ... sf.lit('gender'), df['gender'])).show(truncate=False)
Review Comment:
Yes, it will fail.
<img width="1419" alt="image" src="https://github.com/apache/spark/assets/15246973/9f090017-025c-418d-9512-455eea6c12ea">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on pull request #43168: [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #43168:
URL: https://github.com/apache/spark/pull/43168#issuecomment-1740847611
Now, I will set `SPARK_ANSI_SQL_MODE ` to false.
Let's continue to observe the Ansi workflow daily GA test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1349439434
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
Review Comment:
Because our Python style uses 4 spaces for indentation, I suggest maintaining consistent indentation in the same code snippet (Existing examples and newly added examples.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43168:
URL: https://github.com/apache/spark/pull/43168#discussion_r1349469368
##########
python/pyspark/sql/functions.py:
##########
@@ -11738,15 +11738,29 @@ def create_map(
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2, "female"),
- ... ("Bob", 5, "male")], ("name", "age", "gender"))
+ ... ("Bob", 5, "male")], ("name", "age", "gender"))
Review Comment:
If this is a rule, I suggest fixing it in a separate pr, as there are similar cases to be addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org