You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/07/04 01:12:20 UTC
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/18521
[SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type dispatch in schema verification and improve exception message
## What changes were proposed in this pull request?
**Context**
While reviewing https://github.com/apache/spark/pull/17227, I realised here we type-dispatch per record. The PR itself is fine in terms of performance as is but this prints a prefix, `"obj"` in exception message as below:
```
from pyspark.sql.types import *
schema = StructType([StructField('s', IntegerType(), nullable=False)])
spark.createDataFrame([["1"]], schema)
...
TypeError: obj.s: IntegerType can not accept object '1' in type <type 'str'>
```
I suggested to get rid of this but during investigating this, I realised my approach might bring a performance regression as it is a hot path.
Only for SPARK-19507 and https://github.com/apache/spark/pull/17227, It needs more changes to cleanly get rid of the prefix and I rather decided to fix both issues together.
**Propersal**
This PR tried to
- get rid of per-record type dispatch as we do in many code paths in Scala - SPARK-21296
- Improve error message so that it improves the performance (roughly ~25% improvement) - SPARK-19507
## How was this patch tested?
Manually tested and unit tests were added in `python/pyspark/sql/tests.py`.
Benchmark - codes: https://gist.github.com/HyukjinKwon/c3397469c56cb26c2d7dd521ed0bc5a3
Error message - codes: https://gist.github.com/HyukjinKwon/b1b2c7f65865444c4a8836435100e398
**Before**
Benchmark:
- Results: https://gist.github.com/HyukjinKwon/4a291dab45542106301a0c1abcdca924
Error message
- Results: https://gist.github.com/HyukjinKwon/57b1916395794ce924faa32b14a3fe19
**After**
Benchmark
- Results: https://gist.github.com/HyukjinKwon/21496feecc4a920e50c4e455f836266e
Error message
- Results: https://gist.github.com/HyukjinKwon/7a494e4557fe32a652ce1236e504a395
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark python-type-dispatch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18521.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18521
----
commit a400d5534641813c3ebe75ca553361337766f04b
Author: David Gingrich <da...@textio.com>
Date: 2017-02-28T08:05:00Z
Show field name in _verify_type error
commit 22311f1b4a1a51813904e0be1673b9e8466cac13
Author: hyukjinkwon <gu...@gmail.com>
Date: 2017-07-03T05:41:01Z
Fix default obj parent name issue
commit d7f677830cb423a5da5e428bc3211f348795a2b2
Author: hyukjinkwon <gu...@gmail.com>
Date: 2017-07-03T16:03:50Z
Fix type dispatch
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79141/testReport)** for PR 18521 at commit [`826dcfd`](https://github.com/apache/spark/commit/826dcfd43d6da28ca5b9bfa2d719d6c9ba090e5a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125385778
--- Diff: python/pyspark/sql/tests.py ---
@@ -30,6 +30,19 @@
import functools
import time
import datetime
+import traceback
+
+if sys.version_info[:2] <= (2, 6):
--- End diff --
I will keep in mind.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125386646
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,121 +1249,201 @@ def _infer_schema_type(obj, dataType):
}
-def _verify_type(obj, dataType, nullable=True):
+def _make_type_verifier(dataType, nullable=True, name=None):
"""
Verify the type of obj against dataType, raise a TypeError if they do not match.
Also verify the value of obj against datatype, raise a ValueError if it's not within the allowed
range, e.g. using 128 as ByteType will overflow. Note that, Python float is not checked, so it
will become infinity when cast to Java float if it overflows.
- >>> _verify_type(None, StructType([]))
- >>> _verify_type("", StringType())
- >>> _verify_type(0, LongType())
- >>> _verify_type(list(range(3)), ArrayType(ShortType()))
- >>> _verify_type(set(), ArrayType(StringType())) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(StructType([]))(None)
+ >>> _make_type_verifier(StringType())("")
+ >>> _make_type_verifier(LongType())(0)
+ >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+ >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError:...
- >>> _verify_type({}, MapType(StringType(), IntegerType()))
- >>> _verify_type((), StructType([]))
- >>> _verify_type([], StructType([]))
- >>> _verify_type([1], StructType([])) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+ >>> _make_type_verifier(StructType([]))(())
+ >>> _make_type_verifier(StructType([]))([])
+ >>> _make_type_verifier(StructType([]))([1]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
>>> # Check if numeric values are within the allowed range.
- >>> _verify_type(12, ByteType())
- >>> _verify_type(1234, ByteType()) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType())(12)
+ >>> _make_type_verifier(ByteType())(1234) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type(None, ByteType(), False) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType(), False)(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type([1, None], ArrayType(ShortType(), False)) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(
+ ... ArrayType(ShortType(), False))([1, None]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type({None: 1}, MapType(StringType(), IntegerType()))
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({None: 1})
Traceback (most recent call last):
...
ValueError:...
>>> schema = StructType().add("a", IntegerType()).add("b", StringType(), False)
- >>> _verify_type((1, None), schema) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(schema)((1, None)) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
"""
- if obj is None:
- if nullable:
- return
+
+ if name is None:
+ new_msg = lambda msg: msg
+ new_name = lambda n: "field %s" % n
+ else:
+ new_msg = lambda msg: "%s: %s" % (name, msg)
+ new_name = lambda n: "field %s in %s" % (n, name)
+
+ def verify_nullability(obj):
+ if obj is None:
+ if nullable:
+ return True
+ else:
+ raise ValueError(new_msg("This field is not nullable, but got None"))
else:
- raise ValueError("This field is not nullable, but got None")
+ return False
--- End diff --
how about:
```
def verify_nullability(obj): ...
if isinstance(dataType, StringType):
def verify_string(obj): ...
verify_value = verify_string
elif ...
def verify(obj):
if (verify_nullability(obj)):
return None
verify_value(obj)
return verify
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125386547
--- Diff: python/pyspark/sql/tests.py ---
@@ -30,6 +30,19 @@
import functools
import time
import datetime
+import traceback
+
+if sys.version_info[:2] <= (2, 6):
--- End diff --
Ah, hmm.. it should specifically check Python 2.6 <= as unittest2 is unittest from Python 2.7. To check minor versions, I think we should "extract" or compare with raw string `'2.6'`.
I will clean up inconsistency in another PR about this later if you are okay as is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18521
cc @ueshin and @holdenk who were reviewing it and, @dgingrich the author of that PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79128/testReport)** for PR 18521 at commit [`5b80a8b`](https://github.com/apache/spark/commit/5b80a8b92273e9abf6ce8b28dcd70fbb32d4613c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79141/testReport)** for PR 18521 at commit [`826dcfd`](https://github.com/apache/spark/commit/826dcfd43d6da28ca5b9bfa2d719d6c9ba090e5a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125373417
--- Diff: python/pyspark/sql/tests.py ---
@@ -2620,6 +2635,178 @@ def range_frame_match():
importlib.reload(window)
+
+class TypesTest(unittest.TestCase):
--- End diff --
Many codes, in particular, tests were ported from https://github.com/apache/spark/pull/17227. Many efforts were put here. I think this PR should credit to @dgingrich if there are not many things that should be changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79131/testReport)** for PR 18521 at commit [`9ee8d03`](https://github.com/apache/spark/commit/9ee8d03a951f06a7426f8e924a084d20b25e26d1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79134/testReport)** for PR 18521 at commit [`420b4bf`](https://github.com/apache/spark/commit/420b4bfbe2e0e5f0125df119dd1aed29cfab499a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125382241
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,121 +1249,201 @@ def _infer_schema_type(obj, dataType):
}
-def _verify_type(obj, dataType, nullable=True):
+def _make_type_verifier(dataType, nullable=True, name=None):
"""
Verify the type of obj against dataType, raise a TypeError if they do not match.
Also verify the value of obj against datatype, raise a ValueError if it's not within the allowed
range, e.g. using 128 as ByteType will overflow. Note that, Python float is not checked, so it
will become infinity when cast to Java float if it overflows.
- >>> _verify_type(None, StructType([]))
- >>> _verify_type("", StringType())
- >>> _verify_type(0, LongType())
- >>> _verify_type(list(range(3)), ArrayType(ShortType()))
- >>> _verify_type(set(), ArrayType(StringType())) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(StructType([]))(None)
+ >>> _make_type_verifier(StringType())("")
+ >>> _make_type_verifier(LongType())(0)
+ >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+ >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError:...
- >>> _verify_type({}, MapType(StringType(), IntegerType()))
- >>> _verify_type((), StructType([]))
- >>> _verify_type([], StructType([]))
- >>> _verify_type([1], StructType([])) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+ >>> _make_type_verifier(StructType([]))(())
+ >>> _make_type_verifier(StructType([]))([])
+ >>> _make_type_verifier(StructType([]))([1]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
>>> # Check if numeric values are within the allowed range.
- >>> _verify_type(12, ByteType())
- >>> _verify_type(1234, ByteType()) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType())(12)
+ >>> _make_type_verifier(ByteType())(1234) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type(None, ByteType(), False) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType(), False)(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type([1, None], ArrayType(ShortType(), False)) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(
+ ... ArrayType(ShortType(), False))([1, None]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type({None: 1}, MapType(StringType(), IntegerType()))
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({None: 1})
Traceback (most recent call last):
...
ValueError:...
>>> schema = StructType().add("a", IntegerType()).add("b", StringType(), False)
- >>> _verify_type((1, None), schema) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(schema)((1, None)) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
"""
- if obj is None:
- if nullable:
- return
+
+ if name is None:
+ new_msg = lambda msg: msg
+ new_name = lambda n: "field %s" % n
+ else:
+ new_msg = lambda msg: "%s: %s" % (name, msg)
+ new_name = lambda n: "field %s in %s" % (n, name)
+
+ def verify_nullability(obj):
+ if obj is None:
+ if nullable:
+ return True
+ else:
+ raise ValueError(new_msg("This field is not nullable, but got None"))
else:
- raise ValueError("This field is not nullable, but got None")
+ return False
# StringType can work with any types
if isinstance(dataType, StringType):
- return
+ def verify_string(obj):
+ if verify_nullability(obj):
+ return None
--- End diff --
why a verify method needs to return something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125380790
--- Diff: python/pyspark/sql/tests.py ---
@@ -30,6 +30,19 @@
import functools
import time
import datetime
+import traceback
+
+if sys.version_info[:2] <= (2, 6):
--- End diff --
this is different from how we check python version in other files, is this better than `sys.version >= '3'`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125382032
--- Diff: python/pyspark/sql/session.py ---
@@ -514,17 +514,21 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr
schema = [str(x) for x in data.columns]
data = [r.tolist() for r in data.to_records(index=False)]
- verify_func = _verify_type if verifySchema else lambda _, t: True
if isinstance(schema, StructType):
+ verify_func = _make_type_verifier(schema) if verifySchema else lambda _: True
+
def prepare(obj):
- verify_func(obj, schema)
+ verify_func(obj)
return obj
elif isinstance(schema, DataType):
dataType = schema
schema = StructType().add("value", schema)
+ verify_func = _make_type_verifier(
+ dataType, name="field value") if verifySchema else lambda _: True
--- End diff --
is "field value" useful in the error message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79116/testReport)** for PR 18521 at commit [`d7f6778`](https://github.com/apache/spark/commit/d7f677830cb423a5da5e428bc3211f348795a2b2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125384150
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,121 +1249,201 @@ def _infer_schema_type(obj, dataType):
}
-def _verify_type(obj, dataType, nullable=True):
+def _make_type_verifier(dataType, nullable=True, name=None):
"""
Verify the type of obj against dataType, raise a TypeError if they do not match.
Also verify the value of obj against datatype, raise a ValueError if it's not within the allowed
range, e.g. using 128 as ByteType will overflow. Note that, Python float is not checked, so it
will become infinity when cast to Java float if it overflows.
- >>> _verify_type(None, StructType([]))
- >>> _verify_type("", StringType())
- >>> _verify_type(0, LongType())
- >>> _verify_type(list(range(3)), ArrayType(ShortType()))
- >>> _verify_type(set(), ArrayType(StringType())) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(StructType([]))(None)
+ >>> _make_type_verifier(StringType())("")
+ >>> _make_type_verifier(LongType())(0)
+ >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+ >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError:...
- >>> _verify_type({}, MapType(StringType(), IntegerType()))
- >>> _verify_type((), StructType([]))
- >>> _verify_type([], StructType([]))
- >>> _verify_type([1], StructType([])) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+ >>> _make_type_verifier(StructType([]))(())
+ >>> _make_type_verifier(StructType([]))([])
+ >>> _make_type_verifier(StructType([]))([1]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
>>> # Check if numeric values are within the allowed range.
- >>> _verify_type(12, ByteType())
- >>> _verify_type(1234, ByteType()) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType())(12)
+ >>> _make_type_verifier(ByteType())(1234) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type(None, ByteType(), False) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType(), False)(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type([1, None], ArrayType(ShortType(), False)) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(
+ ... ArrayType(ShortType(), False))([1, None]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type({None: 1}, MapType(StringType(), IntegerType()))
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({None: 1})
Traceback (most recent call last):
...
ValueError:...
>>> schema = StructType().add("a", IntegerType()).add("b", StringType(), False)
- >>> _verify_type((1, None), schema) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(schema)((1, None)) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
"""
- if obj is None:
- if nullable:
- return
+
+ if name is None:
+ new_msg = lambda msg: msg
+ new_name = lambda n: "field %s" % n
+ else:
+ new_msg = lambda msg: "%s: %s" % (name, msg)
+ new_name = lambda n: "field %s in %s" % (n, name)
+
+ def verify_nullability(obj):
+ if obj is None:
+ if nullable:
+ return True
+ else:
+ raise ValueError(new_msg("This field is not nullable, but got None"))
else:
- raise ValueError("This field is not nullable, but got None")
+ return False
# StringType can work with any types
if isinstance(dataType, StringType):
- return
+ def verify_string(obj):
+ if verify_nullability(obj):
+ return None
--- End diff --
```python
def A():
print "a"
def B():
print "a"
return None
def C():
print "a"
return
print(A() is None)
print(A() == B() == C())
```
These are synonyms. I believe this is also about preference - `return` vs `return None` if we should return in the middle of a function. I prefer `return None` as ii might confuse code reading.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79140/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79128/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79140/testReport)** for PR 18521 at commit [`15c575f`](https://github.com/apache/spark/commit/15c575fea9b0adaa58ef1b4c553c9d5bb11990d3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125394827
--- Diff: python/pyspark/sql/tests.py ---
@@ -2620,6 +2635,178 @@ def range_frame_match():
importlib.reload(window)
+
+class TypesTest(unittest.TestCase):
+
+ def test_verify_type_exception_msg(self):
+ def verify_type(obj, dataType, nullable=True, name=None):
+ _make_type_verifier(dataType, nullable, name)(obj)
+
+ name = "test_name"
+ try:
+ verify_type(None, StringType(), nullable=False, name=name)
--- End diff --
shall we use `self.assertRaises` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79131/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125386150
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,121 +1249,201 @@ def _infer_schema_type(obj, dataType):
}
-def _verify_type(obj, dataType, nullable=True):
+def _make_type_verifier(dataType, nullable=True, name=None):
"""
Verify the type of obj against dataType, raise a TypeError if they do not match.
Also verify the value of obj against datatype, raise a ValueError if it's not within the allowed
range, e.g. using 128 as ByteType will overflow. Note that, Python float is not checked, so it
will become infinity when cast to Java float if it overflows.
- >>> _verify_type(None, StructType([]))
- >>> _verify_type("", StringType())
- >>> _verify_type(0, LongType())
- >>> _verify_type(list(range(3)), ArrayType(ShortType()))
- >>> _verify_type(set(), ArrayType(StringType())) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(StructType([]))(None)
+ >>> _make_type_verifier(StringType())("")
+ >>> _make_type_verifier(LongType())(0)
+ >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+ >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError:...
- >>> _verify_type({}, MapType(StringType(), IntegerType()))
- >>> _verify_type((), StructType([]))
- >>> _verify_type([], StructType([]))
- >>> _verify_type([1], StructType([])) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+ >>> _make_type_verifier(StructType([]))(())
+ >>> _make_type_verifier(StructType([]))([])
+ >>> _make_type_verifier(StructType([]))([1]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
>>> # Check if numeric values are within the allowed range.
- >>> _verify_type(12, ByteType())
- >>> _verify_type(1234, ByteType()) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType())(12)
+ >>> _make_type_verifier(ByteType())(1234) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type(None, ByteType(), False) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType(), False)(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type([1, None], ArrayType(ShortType(), False)) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(
+ ... ArrayType(ShortType(), False))([1, None]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type({None: 1}, MapType(StringType(), IntegerType()))
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({None: 1})
Traceback (most recent call last):
...
ValueError:...
>>> schema = StructType().add("a", IntegerType()).add("b", StringType(), False)
- >>> _verify_type((1, None), schema) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(schema)((1, None)) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
"""
- if obj is None:
- if nullable:
- return
+
+ if name is None:
+ new_msg = lambda msg: msg
+ new_name = lambda n: "field %s" % n
+ else:
+ new_msg = lambda msg: "%s: %s" % (name, msg)
+ new_name = lambda n: "field %s in %s" % (n, name)
+
+ def verify_nullability(obj):
+ if obj is None:
+ if nullable:
+ return True
+ else:
+ raise ValueError(new_msg("This field is not nullable, but got None"))
else:
- raise ValueError("This field is not nullable, but got None")
+ return False
# StringType can work with any types
if isinstance(dataType, StringType):
- return
+ def verify_string(obj):
+ if verify_nullability(obj):
+ return None
--- End diff --
ah makes sense, but at least here we are not returning earlier, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79116/testReport)** for PR 18521 at commit [`d7f6778`](https://github.com/apache/spark/commit/d7f677830cb423a5da5e428bc3211f348795a2b2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125373165
--- Diff: python/pyspark/sql/tests.py ---
@@ -2620,6 +2635,178 @@ def range_frame_match():
importlib.reload(window)
+
+class TypesTest(unittest.TestCase):
--- End diff --
Many codes, in particular, tests were ported from https://github.com/apache/spark/pull/17227. Many efforts were put here. I think this PR should credit to @dgingrich.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18521
@cloud-fan, I believe it is ready for another look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125385638
--- Diff: python/pyspark/sql/tests.py ---
@@ -30,6 +30,19 @@
import functools
import time
import datetime
+import traceback
+
+if sys.version_info[:2] <= (2, 6):
--- End diff --
can we follow the existing style here? You can send a PR to update all of them later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125383919
--- Diff: python/pyspark/sql/session.py ---
@@ -514,17 +514,21 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr
schema = [str(x) for x in data.columns]
data = [r.tolist() for r in data.to_records(index=False)]
- verify_func = _verify_type if verifySchema else lambda _, t: True
if isinstance(schema, StructType):
+ verify_func = _make_type_verifier(schema) if verifySchema else lambda _: True
+
def prepare(obj):
- verify_func(obj, schema)
+ verify_func(obj)
return obj
elif isinstance(schema, DataType):
dataType = schema
schema = StructType().add("value", schema)
+ verify_func = _make_type_verifier(
+ dataType, name="field value") if verifySchema else lambda _: True
--- End diff --
I don't think so. It should return `None` I think but I just wanted to avoid other changes for reviewing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125418115
--- Diff: python/pyspark/sql/tests.py ---
@@ -30,6 +30,19 @@
import functools
import time
import datetime
+import traceback
+
+if sys.version_info[:2] <= (2, 6):
--- End diff --
Actually, this should be removed. These are duplicated imports below. I don't know why it was added in that PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18521
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125387151
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,121 +1249,201 @@ def _infer_schema_type(obj, dataType):
}
-def _verify_type(obj, dataType, nullable=True):
+def _make_type_verifier(dataType, nullable=True, name=None):
"""
Verify the type of obj against dataType, raise a TypeError if they do not match.
Also verify the value of obj against datatype, raise a ValueError if it's not within the allowed
range, e.g. using 128 as ByteType will overflow. Note that, Python float is not checked, so it
will become infinity when cast to Java float if it overflows.
- >>> _verify_type(None, StructType([]))
- >>> _verify_type("", StringType())
- >>> _verify_type(0, LongType())
- >>> _verify_type(list(range(3)), ArrayType(ShortType()))
- >>> _verify_type(set(), ArrayType(StringType())) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(StructType([]))(None)
+ >>> _make_type_verifier(StringType())("")
+ >>> _make_type_verifier(LongType())(0)
+ >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+ >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError:...
- >>> _verify_type({}, MapType(StringType(), IntegerType()))
- >>> _verify_type((), StructType([]))
- >>> _verify_type([], StructType([]))
- >>> _verify_type([1], StructType([])) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+ >>> _make_type_verifier(StructType([]))(())
+ >>> _make_type_verifier(StructType([]))([])
+ >>> _make_type_verifier(StructType([]))([1]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
>>> # Check if numeric values are within the allowed range.
- >>> _verify_type(12, ByteType())
- >>> _verify_type(1234, ByteType()) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType())(12)
+ >>> _make_type_verifier(ByteType())(1234) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type(None, ByteType(), False) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType(), False)(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type([1, None], ArrayType(ShortType(), False)) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(
+ ... ArrayType(ShortType(), False))([1, None]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type({None: 1}, MapType(StringType(), IntegerType()))
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({None: 1})
Traceback (most recent call last):
...
ValueError:...
>>> schema = StructType().add("a", IntegerType()).add("b", StringType(), False)
- >>> _verify_type((1, None), schema) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(schema)((1, None)) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
"""
- if obj is None:
- if nullable:
- return
+
+ if name is None:
+ new_msg = lambda msg: msg
+ new_name = lambda n: "field %s" % n
+ else:
+ new_msg = lambda msg: "%s: %s" % (name, msg)
+ new_name = lambda n: "field %s in %s" % (n, name)
+
+ def verify_nullability(obj):
+ if obj is None:
+ if nullable:
+ return True
+ else:
+ raise ValueError(new_msg("This field is not nullable, but got None"))
else:
- raise ValueError("This field is not nullable, but got None")
+ return False
--- End diff --
sounds good. Will give a shot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79140/testReport)** for PR 18521 at commit [`15c575f`](https://github.com/apache/spark/commit/15c575fea9b0adaa58ef1b4c553c9d5bb11990d3).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125383823
--- Diff: python/pyspark/sql/tests.py ---
@@ -30,6 +30,19 @@
import functools
import time
import datetime
+import traceback
+
+if sys.version_info[:2] <= (2, 6):
--- End diff --
I think it is a preference. I don't think either is particularly better. Just per documentation. it sounds `version_info` is preferred - https://docs.python.org/2/library/sys.html#sys.version
> Do not extract version information out of it, rather, use `version_info` and the functions
although we are not "extract"ing anyway ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79134/testReport)** for PR 18521 at commit [`420b4bf`](https://github.com/apache/spark/commit/420b4bfbe2e0e5f0125df119dd1aed29cfab499a).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18521
LGTM, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18521
cc @cloud-fan who I believe reviewed my related few PRs before and @davies who I believe is used to this code path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79131/testReport)** for PR 18521 at commit [`9ee8d03`](https://github.com/apache/spark/commit/9ee8d03a951f06a7426f8e924a084d20b25e26d1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class DataTypeVerificationTests(unittest.TestCase):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125385500
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,121 +1249,201 @@ def _infer_schema_type(obj, dataType):
}
-def _verify_type(obj, dataType, nullable=True):
+def _make_type_verifier(dataType, nullable=True, name=None):
"""
Verify the type of obj against dataType, raise a TypeError if they do not match.
Also verify the value of obj against datatype, raise a ValueError if it's not within the allowed
range, e.g. using 128 as ByteType will overflow. Note that, Python float is not checked, so it
will become infinity when cast to Java float if it overflows.
- >>> _verify_type(None, StructType([]))
- >>> _verify_type("", StringType())
- >>> _verify_type(0, LongType())
- >>> _verify_type(list(range(3)), ArrayType(ShortType()))
- >>> _verify_type(set(), ArrayType(StringType())) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(StructType([]))(None)
+ >>> _make_type_verifier(StringType())("")
+ >>> _make_type_verifier(LongType())(0)
+ >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+ >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError:...
- >>> _verify_type({}, MapType(StringType(), IntegerType()))
- >>> _verify_type((), StructType([]))
- >>> _verify_type([], StructType([]))
- >>> _verify_type([1], StructType([])) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+ >>> _make_type_verifier(StructType([]))(())
+ >>> _make_type_verifier(StructType([]))([])
+ >>> _make_type_verifier(StructType([]))([1]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
>>> # Check if numeric values are within the allowed range.
- >>> _verify_type(12, ByteType())
- >>> _verify_type(1234, ByteType()) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType())(12)
+ >>> _make_type_verifier(ByteType())(1234) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type(None, ByteType(), False) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(ByteType(), False)(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type([1, None], ArrayType(ShortType(), False)) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(
+ ... ArrayType(ShortType(), False))([1, None]) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
- >>> _verify_type({None: 1}, MapType(StringType(), IntegerType()))
+ >>> _make_type_verifier(MapType(StringType(), IntegerType()))({None: 1})
Traceback (most recent call last):
...
ValueError:...
>>> schema = StructType().add("a", IntegerType()).add("b", StringType(), False)
- >>> _verify_type((1, None), schema) # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> _make_type_verifier(schema)((1, None)) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
ValueError:...
"""
- if obj is None:
- if nullable:
- return
+
+ if name is None:
--- End diff --
This looks a bit odds but I could not figure out a shorter and cleaner way ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79134/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18521
**[Test build #79128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79128/testReport)** for PR 18521 at commit [`5b80a8b`](https://github.com/apache/spark/commit/5b80a8b92273e9abf6ce8b28dcd70fbb32d4613c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125399210
--- Diff: python/pyspark/sql/tests.py ---
@@ -2620,6 +2635,178 @@ def range_frame_match():
importlib.reload(window)
+
+class TypesTest(unittest.TestCase):
+
+ def test_verify_type_exception_msg(self):
+ def verify_type(obj, dataType, nullable=True, name=None):
+ _make_type_verifier(dataType, nullable, name)(obj)
+
+ name = "test_name"
+ try:
+ verify_type(None, StringType(), nullable=False, name=name)
--- End diff --
Will also clean up the codes here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-reco...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18521#discussion_r125384834
--- Diff: python/pyspark/sql/session.py ---
@@ -514,17 +514,21 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr
schema = [str(x) for x in data.columns]
data = [r.tolist() for r in data.to_records(index=False)]
- verify_func = _verify_type if verifySchema else lambda _, t: True
if isinstance(schema, StructType):
+ verify_func = _make_type_verifier(schema) if verifySchema else lambda _: True
+
def prepare(obj):
- verify_func(obj, schema)
+ verify_func(obj)
return obj
elif isinstance(schema, DataType):
dataType = schema
schema = StructType().add("value", schema)
+ verify_func = _make_type_verifier(
+ dataType, name="field value") if verifySchema else lambda _: True
--- End diff --
Oh, wait you mean `field value`. Yes, this is printed as is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79141/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18521: [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18521
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79116/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org