You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by AlexanderKoryagin <gi...@git.apache.org> on 2018/09/27 10:04:28 UTC
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
GitHub user AlexanderKoryagin opened a pull request:
https://github.com/apache/spark/pull/22568
[SPARK-23401][PYTHON][TESTS] Add more data types for PandasUDFTests
## What changes were proposed in this pull request?
Add more data types for Pandas UDF Tests for PySpark SQL
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlexanderKoryagin/spark new_types_for_pandas_udf_tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22568.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22568
----
commit 4a57833b5fd3f8b36085bbee70bcdf9e94a0a476
Author: Aleksandr Koriagin <al...@...>
Date: 2018-09-27T09:53:19Z
Add more test cases for supported and unsupported types for PandasUDFTests
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221158026
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
--- End diff --
ok not a big deal. I just wanted to make the lines less. Can you simply the last parenthese up then?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96691/testReport)** for PR 22568 at commit [`b27e0a1`](https://github.com/apache/spark/commit/b27e0a1532595df58f9c53304e8ba793ad2a35e9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96744/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220948793
--- Diff: python/pyspark/sql/tests.py ---
@@ -5714,24 +5755,31 @@ def test_wrong_args(self):
pandas_udf(lambda x, y: x, DoubleType(), PandasUDFType.SCALAR))
def test_unsupported_types(self):
+ from distutils.version import LooseVersion
+ import pyarrow as pa
from pyspark.sql.functions import pandas_udf, PandasUDFType
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("map", MapType(StringType(), IntegerType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*MapType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("arr_ts", ArrayType(TimestampType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*ArrayType.*TimestampType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
+ common_err_msg = 'Invalid returnType.*grouped map Pandas UDF.*'
+ unsupported_types = [
+ StructField('map', MapType(StringType(), IntegerType())),
+ StructField('arr_ts', ArrayType(TimestampType())),
+ StructField('null', NullType()),
+ ]
+
+ # TODO: Remove this if-statement once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ unsupported_types.append(
+ StructField('bin', BinaryType())
--- End diff --
Likewise, let's just
```
unsupported_types.append(StructField('bin', BinaryType()))
```
or
```
unsupported_types += [StructField('bin', BinaryType())]
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220948302
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
--- End diff --
I understood why you did this but I think we can just do like:
```python
values = [
1, 2, 3,
4, 5, 1.1,
...
]
output_schema = StructType([
StructField('id', IntegerType()), StructField('byte', ByteType()), StructField('short', ShortType()),
StructField('int', IntegerType()), StructField('long', LongType()), StructField('float', FloatType()),
...
])
```
Let's just keep the original way and make it simple.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96690/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221425094
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,78 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ values = [
+ 1, 2, 3,
+ 4, 5, 1.1,
+ 2.2, Decimal(1.123),
+ [1, 2, 2], True, 'hello'
+ ]
+ output_fields = [
+ ('id', IntegerType()), ('byte', ByteType()), ('short', ShortType()),
+ ('int', IntegerType()), ('long', LongType()), ('float', FloatType()),
+ ('double', DoubleType()), ('decim', DecimalType(10, 3)),
+ ('array', ArrayType(IntegerType())), ('bool', BooleanType()), ('str', StringType())
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to variables above once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ values.append(bytearray([0x01, 0x02]))
+ output_fields.append(('bin', BinaryType()))
+ output_schema = StructType([StructField(*x) for x in output_fields])
+ df = self.spark.createDataFrame([values], schema=output_schema)
+
+ # Different forms of group map pandas UDF, results of these are the same
udf1 = pandas_udf(
- lambda pdf: pdf.assign(v1=pdf.v * pdf.id * 1.0, v2=pdf.v + pdf.id),
+ lambda pdf: pdf.assign(
+ byte=pdf.byte * 2,
+ short=pdf.short * 2,
+ int=pdf.int * 2,
+ long=pdf.long * 2,
+ float=pdf.float * 2,
+ double=pdf.double * 2,
+ decim=pdf.decim * 2,
+ bool=False if pdf.bool else True,
+ str=pdf.str + 'there',
--- End diff --
For now, I think you can just pass the array as is to make the tests passed. The problem you mentioned sounds still orthogonal with Pandas UDF functionaility itself.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96747/testReport)** for PR 22568 at commit [`90a4b1f`](https://github.com/apache/spark/commit/90a4b1f340de934ebcf8378d3b6a71e6efcc3093).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96744/testReport)** for PR 22568 at commit [`20e360e`](https://github.com/apache/spark/commit/20e360e93f3ae89b9d9dc8b3ef8bd2e69d0b7e6a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96747/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220880080
--- Diff: python/pyspark/sql/tests.py ---
@@ -5715,23 +5752,29 @@ def test_wrong_args(self):
def test_unsupported_types(self):
from pyspark.sql.functions import pandas_udf, PandasUDFType
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("map", MapType(StringType(), IntegerType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*MapType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
-
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("arr_ts", ArrayType(TimestampType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*ArrayType.*TimestampType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
+ # type, error message regexp
+ unsupported_types_with_msg = (
--- End diff --
I think one `Invalid returnType.*grouped map Pandas UDF.*` is good enough.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96690/testReport)** for PR 22568 at commit [`4a57833`](https://github.com/apache/spark/commit/4a57833b5fd3f8b36085bbee70bcdf9e94a0a476).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96755/testReport)** for PR 22568 at commit [`f5b6ac8`](https://github.com/apache/spark/commit/f5b6ac8de222f7f6a903d49dcc3b4fee5d4172d8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22568
Please feel free to merge https://github.com/AlexanderKoryagin/spark/pull/1 to your branch. Should be good to go then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96756/testReport)** for PR 22568 at commit [`5b8ca68`](https://github.com/apache/spark/commit/5b8ca68f608b6724d1227798a3b008a7010a2d54).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96756/testReport)** for PR 22568 at commit [`5b8ca68`](https://github.com/apache/spark/commit/5b8ca68f608b6724d1227798a3b008a7010a2d54).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220943714
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
--- End diff --
tiny nit: I would just
```python
input_values_with_schema += [bytearray([0x01, 0x02]), StructField('bin', BinaryType())]
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220878993
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,69 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
+ from decimal import Decimal
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = (
--- End diff --
can we avoid vertical align
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220944429
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
+ )
+
+ values = [[x[0] for x in input_values_with_schema]]
+ output_schema = StructType([x[1] for x in input_values_with_schema])
+ df = self.spark.createDataFrame(values, schema=output_schema)
+
+ # Different forms of group map pandas UDF, results of these are the same
udf1 = pandas_udf(
- lambda pdf: pdf.assign(v1=pdf.v * pdf.id * 1.0, v2=pdf.v + pdf.id),
+ lambda pdf: pdf.assign(
+ decim=pdf.decim + pdf.decim,
+ double=pdf.double + pdf.float,
+ byte=pdf.byte + 1,
+ long=pdf.byte + pdf.int + pdf.long + pdf.short,
--- End diff --
I would get rid of those calculations with different types to make it easier to read.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221158302
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
--- End diff --
BTW, it's 100 characters in Spark unlike PEP8's 80.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220884563
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,69 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
+ from decimal import Decimal
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96756/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221183502
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
--- End diff --
Are you sure that keeping the original way is more important then making code more readable and maintainable?
IMHO with such way it will be hard to visually match values with types:
```python
values = [
1, 2, 3, 4, 5, 1.1, 2.2, Decimal(1.123),
[1, 2, 2], True, 'hello'
]
output_schema = [
StructField('id', IntegerType()), StructField('byte', ByteType()),
StructField('short', ShortType()), StructField('int', IntegerType()),
StructField('long', LongType()), StructField('float', FloatType()),
StructField('double', DoubleType()), StructField('decim', DecimalType(10, 3)),
StructField('array', ArrayType(IntegerType())), StructField('bool', BooleanType()),
StructField('str', StringType())
]
```
But anyway, if you insists , I can change
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22568
Merged to master and branch-2.4.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221186971
--- Diff: python/pyspark/sql/tests.py ---
@@ -5714,24 +5755,31 @@ def test_wrong_args(self):
pandas_udf(lambda x, y: x, DoubleType(), PandasUDFType.SCALAR))
def test_unsupported_types(self):
+ from distutils.version import LooseVersion
+ import pyarrow as pa
from pyspark.sql.functions import pandas_udf, PandasUDFType
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("map", MapType(StringType(), IntegerType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*MapType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("arr_ts", ArrayType(TimestampType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*ArrayType.*TimestampType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
+ common_err_msg = 'Invalid returnType.*grouped map Pandas UDF.*'
+ unsupported_types = [
+ StructField('map', MapType(StringType(), IntegerType())),
+ StructField('arr_ts', ArrayType(TimestampType())),
+ StructField('null', NullType()),
+ ]
+
+ # TODO: Remove this if-statement once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ unsupported_types.append(
+ StructField('bin', BinaryType())
+ )
+
+ for unsupported_type in unsupported_types:
+ schema = StructType([
+ StructField('id', LongType(), True),
+ unsupported_type
+ ])
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221186913
--- Diff: python/pyspark/sql/tests.py ---
@@ -5714,24 +5755,31 @@ def test_wrong_args(self):
pandas_udf(lambda x, y: x, DoubleType(), PandasUDFType.SCALAR))
def test_unsupported_types(self):
+ from distutils.version import LooseVersion
+ import pyarrow as pa
from pyspark.sql.functions import pandas_udf, PandasUDFType
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("map", MapType(StringType(), IntegerType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*MapType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("arr_ts", ArrayType(TimestampType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*ArrayType.*TimestampType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
+ common_err_msg = 'Invalid returnType.*grouped map Pandas UDF.*'
+ unsupported_types = [
+ StructField('map', MapType(StringType(), IntegerType())),
+ StructField('arr_ts', ArrayType(TimestampType())),
+ StructField('null', NullType()),
+ ]
+
+ # TODO: Remove this if-statement once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ unsupported_types.append(
+ StructField('bin', BinaryType())
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221186707
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22568
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on the issue:
https://github.com/apache/spark/pull/22568
@HyukjinKwon
Just noticed strange behavior, that can be reproduced with a test below placed under `python.pyspark.sql.tests.GroupedMapPandasUDFTests`.
Is it a bug or some feature?
```python
def test_supported_types_array(self):
from pyspark.sql.functions import pandas_udf, PandasUDFType
schema = StructType([
StructField('id', IntegerType()),
StructField('array', ArrayType(IntegerType()))
])
df = self.spark.createDataFrame(
[[1, [1, 2, 3]]], schema=schema
)
udf1 = pandas_udf(
lambda pdf: pdf.assign(array=pdf.array * 2),
schema,
PandasUDFType.GROUPED_MAP
)
result1 = df.groupby('id').apply(udf1).sort('id').toPandas()
expected1 = df.toPandas().groupby('id').apply(udf1.func).reset_index(drop=True)
self.assertPandasEqual(expected1, result1)
```
Here is output:
```
python/pyspark/sql/tests.py:244: in assertPandasEqual
self.assertTrue(expected.equals(result), msg=msg)
E AssertionError: DataFrames are not equal:
E
E Expected:
E id array
E 0 1 [1, 2, 3, 1, 2, 3]
E id int32
E array object
E dtype: object
E
E Result:
E id array
E 0 1 [2, 4, 6]
E id int32
E array object
E dtype: object
```
You can see that behavior of `array=pdf.array * 2` different for `result1` and `expected1`:
```
result1 = df.groupby('id').apply(udf1).sort('id').toPandas()
>> [2, 4, 6]
expected1 = df.toPandas().groupby('id').apply(udf1.func).reset_index(drop=True)
>> [1, 2, 3, 1, 2, 3]
```
Default Python behavior is:
```
[1, 2, 3] * 2
>> [1, 2, 3, 1, 2, 3]
```
Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96755/testReport)** for PR 22568 at commit [`f5b6ac8`](https://github.com/apache/spark/commit/f5b6ac8de222f7f6a903d49dcc3b4fee5d4172d8).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22568
re: https://github.com/apache/spark/pull/22568#issuecomment-425352709
That's because within Pandas UDF it's `numpy.ndarray` which allows vectorized operations whereas the latter case is a Python object list (when Arrow optimization is disabled by default).
Try:
```
from pyspark.sql.functions import pandas_udf, PandasUDFType
from pyspark.sql.types import *
schema = StructType([
StructField('id', IntegerType()),
StructField('array', ArrayType(IntegerType()))])
df = spark.createDataFrame([[1, [1, 2, 3]]], schema=schema)
def foo(pdf):
print type(pdf.array[0])
return pdf.assign(array=pdf.array * 2)
udf1 = pandas_udf(foo, schema, PandasUDFType.GROUPED_MAP)
result1 = df.groupby('id').apply(udf1).sort('id').toPandas()
print(type(result1.array[0]))
```
There's difference about type conversion details when Arrow is enabled/disabled. You will have the same results if you enable `spark.sql.execution.arrow.enabled` (which means enabling Arrow optimization for `toPandas` and `createDataFrame`).
The type conversion (powered by Arrow) is not exactly matched to original Pnadas PySpark's conversion. We should match both. For now, I think matching it to `numpy.ndarray` makes more sense when `spark.sql.execution.arrow.enabled` is disabled from a cursory look. Probably we should fix array conversion from Array -> a list to array to numpy array (from JVM to Python). My guys say it needs some more discussion and investigation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220879251
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,69 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
+ from decimal import Decimal
--- End diff --
not a big deal but
```
from decimal import Decimal
from pyspark.sql.functions import pandas_udf, PandasUDFType
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96692/testReport)** for PR 22568 at commit [`53ff750`](https://github.com/apache/spark/commit/53ff750456057098b029a801266818fc7204cf79).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221186790
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
+ )
+
+ values = [[x[0] for x in input_values_with_schema]]
+ output_schema = StructType([x[1] for x in input_values_with_schema])
+ df = self.spark.createDataFrame(values, schema=output_schema)
+
+ # Different forms of group map pandas UDF, results of these are the same
udf1 = pandas_udf(
- lambda pdf: pdf.assign(v1=pdf.v * pdf.id * 1.0, v2=pdf.v + pdf.id),
+ lambda pdf: pdf.assign(
+ decim=pdf.decim + pdf.decim,
+ double=pdf.double + pdf.float,
+ byte=pdf.byte + 1,
+ long=pdf.byte + pdf.int + pdf.long + pdf.short,
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96692/testReport)** for PR 22568 at commit [`53ff750`](https://github.com/apache/spark/commit/53ff750456057098b029a801266818fc7204cf79).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96812/testReport)** for PR 22568 at commit [`2d6fb62`](https://github.com/apache/spark/commit/2d6fb62cae94f00d7466514bfadee1797e08d34c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96812/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221529918
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,78 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ values = [
+ 1, 2, 3,
+ 4, 5, 1.1,
+ 2.2, Decimal(1.123),
+ [1, 2, 2], True, 'hello'
+ ]
+ output_fields = [
+ ('id', IntegerType()), ('byte', ByteType()), ('short', ShortType()),
+ ('int', IntegerType()), ('long', LongType()), ('float', FloatType()),
+ ('double', DoubleType()), ('decim', DecimalType(10, 3)),
+ ('array', ArrayType(IntegerType())), ('bool', BooleanType()), ('str', StringType())
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to variables above once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ values.append(bytearray([0x01, 0x02]))
+ output_fields.append(('bin', BinaryType()))
+ output_schema = StructType([StructField(*x) for x in output_fields])
+ df = self.spark.createDataFrame([values], schema=output_schema)
+
+ # Different forms of group map pandas UDF, results of these are the same
udf1 = pandas_udf(
- lambda pdf: pdf.assign(v1=pdf.v * pdf.id * 1.0, v2=pdf.v + pdf.id),
+ lambda pdf: pdf.assign(
+ byte=pdf.byte * 2,
+ short=pdf.short * 2,
+ int=pdf.int * 2,
+ long=pdf.long * 2,
+ float=pdf.float * 2,
+ double=pdf.double * 2,
+ decim=pdf.decim * 2,
+ bool=False if pdf.bool else True,
+ str=pdf.str + 'there',
--- End diff --
Thanks, merged
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22568
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96747/testReport)** for PR 22568 at commit [`90a4b1f`](https://github.com/apache/spark/commit/90a4b1f340de934ebcf8378d3b6a71e6efcc3093).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96812/testReport)** for PR 22568 at commit [`2d6fb62`](https://github.com/apache/spark/commit/2d6fb62cae94f00d7466514bfadee1797e08d34c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96690/testReport)** for PR 22568 at commit [`4a57833`](https://github.com/apache/spark/commit/4a57833b5fd3f8b36085bbee70bcdf9e94a0a476).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221194037
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
--- End diff --
Eh, how about:
```
values = [
1, 2, 3,
4, 5, 1.1,
...
]
output_fields = [
('id', IntegerType()), ('byte', ByteType()), ('short', ShortType()),
('int', IntegerType()), ('long', LongType()), ('float', FloatType()),
...
])
output_schema = StructType(map(lambda x: StructField(*x), output_fields))
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221203078
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
--- End diff --
Thanks, done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96755/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220948939
--- Diff: python/pyspark/sql/tests.py ---
@@ -5714,24 +5755,31 @@ def test_wrong_args(self):
pandas_udf(lambda x, y: x, DoubleType(), PandasUDFType.SCALAR))
def test_unsupported_types(self):
+ from distutils.version import LooseVersion
+ import pyarrow as pa
from pyspark.sql.functions import pandas_udf, PandasUDFType
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("map", MapType(StringType(), IntegerType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*MapType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
- schema = StructType(
- [StructField("id", LongType(), True),
- StructField("arr_ts", ArrayType(TimestampType()), True)])
- with QuietTest(self.sc):
- with self.assertRaisesRegexp(
- NotImplementedError,
- 'Invalid returnType.*grouped map Pandas UDF.*ArrayType.*TimestampType'):
- pandas_udf(lambda x: x, schema, PandasUDFType.GROUPED_MAP)
+ common_err_msg = 'Invalid returnType.*grouped map Pandas UDF.*'
+ unsupported_types = [
+ StructField('map', MapType(StringType(), IntegerType())),
+ StructField('arr_ts', ArrayType(TimestampType())),
+ StructField('null', NullType()),
+ ]
+
+ # TODO: Remove this if-statement once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ unsupported_types.append(
+ StructField('bin', BinaryType())
+ )
+
+ for unsupported_type in unsupported_types:
+ schema = StructType([
+ StructField('id', LongType(), True),
+ unsupported_type
+ ])
--- End diff --
I think we can make this inlined as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96691/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221425061
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,78 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ values = [
+ 1, 2, 3,
+ 4, 5, 1.1,
+ 2.2, Decimal(1.123),
+ [1, 2, 2], True, 'hello'
+ ]
+ output_fields = [
+ ('id', IntegerType()), ('byte', ByteType()), ('short', ShortType()),
+ ('int', IntegerType()), ('long', LongType()), ('float', FloatType()),
+ ('double', DoubleType()), ('decim', DecimalType(10, 3)),
+ ('array', ArrayType(IntegerType())), ('bool', BooleanType()), ('str', StringType())
--- End diff --
For now, I think you can just pass the array as is.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96744/testReport)** for PR 22568 at commit [`20e360e`](https://github.com/apache/spark/commit/20e360e93f3ae89b9d9dc8b3ef8bd2e69d0b7e6a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r220884525
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,69 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
+ from decimal import Decimal
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = (
--- End diff --
Done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Posted by AlexanderKoryagin <gi...@git.apache.org>.
Github user AlexanderKoryagin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22568#discussion_r221156223
--- Diff: python/pyspark/sql/tests.py ---
@@ -5525,32 +5525,73 @@ def data(self):
.withColumn("v", explode(col('vs'))).drop('vs')
def test_supported_types(self):
- from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col
- df = self.data.withColumn("arr", array(col("id")))
+ from decimal import Decimal
+ from distutils.version import LooseVersion
+ import pyarrow as pa
+ from pyspark.sql.functions import pandas_udf, PandasUDFType
- # Different forms of group map pandas UDF, results of these are the same
+ input_values_with_schema = [
+ (1, StructField('id', IntegerType())),
+ (2, StructField('byte', ByteType())),
+ (3, StructField('short', ShortType())),
+ (4, StructField('int', IntegerType())),
+ (5, StructField('long', LongType())),
+ (1.1, StructField('float', FloatType())),
+ (2.2, StructField('double', DoubleType())),
+ (Decimal(1.123), StructField('decim', DecimalType(10, 3))),
+ ([1, 2, 3], StructField('array', ArrayType(IntegerType()))),
+ (True, StructField('bool', BooleanType())),
+ ('hello', StructField('str', StringType())),
+ ]
- output_schema = StructType(
- [StructField('id', LongType()),
- StructField('v', IntegerType()),
- StructField('arr', ArrayType(LongType())),
- StructField('v1', DoubleType()),
- StructField('v2', LongType())])
+ # TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"):
+ input_values_with_schema.append(
+ (bytearray([0x01, 0x02]), StructField('bin', BinaryType()))
--- End diff --
Thanks for comment! But I think it is better to use native list method `append` here instead of creating a new list with only one value in it and concatenate it with another.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96692/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22568
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22568: [SPARK-23401][PYTHON][TESTS] Add more data types for Pan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22568
**[Test build #96691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96691/testReport)** for PR 22568 at commit [`b27e0a1`](https://github.com/apache/spark/commit/b27e0a1532595df58f9c53304e8ba793ad2a35e9).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org