You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/05/13 05:59:16 UTC
[spark] branch branch-3.1 updated: [SPARK-35382][PYTHON] Fix lambda
variable name issues in nested DataFrame functions in Python APIs
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 67e4c94 [SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs
67e4c94 is described below
commit 67e4c94d1d393766e5ca009b6475db5b2fb034bb
Author: Takuya UESHIN <ue...@databricks.com>
AuthorDate: Thu May 13 14:58:01 2021 +0900
[SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs
### What changes were proposed in this pull request?
This PR fixes the same issue as #32424.
```py
from pyspark.sql.functions import flatten, struct, transform
df = spark.sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
df.select(flatten(
transform(
"numbers",
lambda number: transform(
"letters",
lambda letter: struct(number.alias("n"), letter.alias("l"))
)
)
).alias("zipped")).show(truncate=False)
```
**Before:**
```
+------------------------------------------------------------------------+
|zipped |
+------------------------------------------------------------------------+
|[{a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}]|
+------------------------------------------------------------------------+
```
**After:**
```
+------------------------------------------------------------------------+
|zipped |
+------------------------------------------------------------------------+
|[{1, a}, {1, b}, {1, c}, {2, a}, {2, b}, {2, c}, {3, a}, {3, b}, {3, c}]|
+------------------------------------------------------------------------+
```
### Why are the changes needed?
To produce the correct results.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes the results to be correct as mentioned above.
### How was this patch tested?
Added a unit test as well as manually.
Closes #32523 from ueshin/issues/SPARK-35382/nested_higher_order_functions.
Authored-by: Takuya UESHIN <ue...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 17b59a9970a0079ac9225de52247a1de4772c1fa)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/sql/functions.py | 5 ++++-
python/pyspark/sql/tests/test_functions.py | 22 ++++++++++++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 51ab9c1..2f1857d 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -4153,7 +4153,10 @@ def _create_lambda(f):
argnames = ["x", "y", "z"]
args = [
- _unresolved_named_lambda_variable(arg) for arg in argnames[: len(parameters)]
+ _unresolved_named_lambda_variable(
+ expressions.UnresolvedNamedLambdaVariable.freshVarName(arg)
+ )
+ for arg in argnames[: len(parameters)]
]
result = f(*args)
diff --git a/python/pyspark/sql/tests/test_functions.py b/python/pyspark/sql/tests/test_functions.py
index 053164a..8ccc051 100644
--- a/python/pyspark/sql/tests/test_functions.py
+++ b/python/pyspark/sql/tests/test_functions.py
@@ -491,6 +491,28 @@ class FunctionsTests(ReusedSQLTestCase):
with self.assertRaises(ValueError):
transform(col("foo"), lambda x: 1)
+ def test_nested_higher_order_function(self):
+ # SPARK-35382: lambda vars must be resolved properly in nested higher order functions
+ from pyspark.sql.functions import flatten, struct, transform
+
+ df = self.spark.sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
+
+ actual = df.select(flatten(
+ transform(
+ "numbers",
+ lambda number: transform(
+ "letters",
+ lambda letter: struct(number.alias("n"), letter.alias("l"))
+ )
+ )
+ )).first()[0]
+
+ expected = [(1, "a"), (1, "b"), (1, "c"),
+ (2, "a"), (2, "b"), (2, "c"),
+ (3, "a"), (3, "b"), (3, "c")]
+
+ self.assertEquals(actual, expected)
+
def test_window_functions(self):
df = self.spark.createDataFrame([(1, "1"), (2, "2"), (1, "2"), (1, "2")], ["key", "value"])
w = Window.partitionBy("value").orderBy("key")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org