You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by cu...@apache.org on 2018/09/06 17:17:40 UTC
spark git commit: [SPARK-25072][PYSPARK] Forbid extra value for
custom Row
Repository: spark
Updated Branches:
refs/heads/master 3b6591b0b -> c84bc40d7
[SPARK-25072][PYSPARK] Forbid extra value for custom Row
## What changes were proposed in this pull request?
Add value length check in `_create_row`, forbid extra value for custom Row in PySpark.
## How was this patch tested?
New UT in pyspark-sql
Closes #22140 from xuanyuanking/SPARK-25072.
Lead-authored-by: liyuanjian <li...@baidu.com>
Co-authored-by: Yuanjian Li <xy...@gmail.com>
Signed-off-by: Bryan Cutler <cu...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c84bc40d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c84bc40d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c84bc40d
Branch: refs/heads/master
Commit: c84bc40d7f33c71eca1c08f122cd60517f34c1f8
Parents: 3b6591b
Author: liyuanjian <li...@baidu.com>
Authored: Thu Sep 6 10:17:29 2018 -0700
Committer: Bryan Cutler <cu...@gmail.com>
Committed: Thu Sep 6 10:17:29 2018 -0700
----------------------------------------------------------------------
python/pyspark/sql/tests.py | 4 ++++
python/pyspark/sql/types.py | 3 +++
2 files changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c84bc40d/python/pyspark/sql/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index 81c0af0..6d9d636 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -277,6 +277,10 @@ class DataTypeTests(unittest.TestCase):
struct_field = StructField("a", IntegerType())
self.assertRaises(TypeError, struct_field.typeName)
+ def test_invalid_create_row(self):
+ row_class = Row("c1", "c2")
+ self.assertRaises(ValueError, lambda: row_class(1, 2, 3))
+
class SQLTests(ReusedSQLTestCase):
http://git-wip-us.apache.org/repos/asf/spark/blob/c84bc40d/python/pyspark/sql/types.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 0b61707..ce1d004 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -1500,6 +1500,9 @@ class Row(tuple):
# let object acts like class
def __call__(self, *args):
"""create new Row object"""
+ if len(args) > len(self):
+ raise ValueError("Can not create Row with fields %s, expected %d values "
+ "but got %s" % (self, len(self), args))
return _create_row(self, args)
def __getitem__(self, item):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org