You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by BryanCutler <gi...@git.apache.org> on 2018/03/03 00:36:03 UTC
[GitHub] spark pull request #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support...
GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/20725
[WIP][SPARK-23555][PYTHON] Add BinaryType support for Arrow
## What changes were proposed in this pull request?
Adding `BinaryType` support for Arrow in pyspark.
## How was this patch tested?
Additional unit tests in pyspark for code paths that use Arrow
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark arrow-binary-type-support-SPARK-23555
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20725.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20725
----
commit afcb0d5608c17d2fc004a0b6d4af4573abca4e4b
Author: Bryan Cutler <cu...@...>
Date: 2018-03-03T00:32:45Z
added support for binary type, not currently working due to arrow error
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/20725
@BryanCutler -- the ubuntu PRB passed w/pyarrow 0.10.0!
https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/84/console
i'm still twiddling some bits on getting the anaconda env to properly stick, but that doesn't impact your changes here. i'm going to say that this should be g2g.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20725
**[Test build #94838 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94838/testReport)** for PR 20725 at commit [`461c326`](https://github.com/apache/spark/commit/461c326f00d68a350a1b5c0f7b644f2871ee0a85).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/20725
https://rise.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/79/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20725
**[Test build #93854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93854/testReport)** for PR 20725 at commit [`45eea0c`](https://github.com/apache/spark/commit/45eea0c99f738cebd97bc643ad413a5f3957a045).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20725
**[Test build #94838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94838/testReport)** for PR 20725 at commit [`461c326`](https://github.com/apache/spark/commit/461c326f00d68a350a1b5c0f7b644f2871ee0a85).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94838/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20725: [SPARK-23555][PYTHON] Add BinaryType support for ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20725#discussion_r211715497
--- Diff: python/pyspark/sql/types.py ---
@@ -1597,6 +1598,12 @@ def to_arrow_type(dt):
arrow_type = pa.decimal128(dt.precision, dt.scale)
elif type(dt) == StringType:
arrow_type = pa.string()
+ elif type(dt) == BinaryType:
+ # TODO: remove version check once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ raise TypeError("Unsupported type in conversion to Arrow: " + str(dt) +
--- End diff --
Could you document all the features that require pyarrow 0.10?
Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/20725
https://rise.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/80/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20725: [SPARK-23555][PYTHON] Add BinaryType support for ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20725
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/20725
ugh hang on. lemme fix that crap. (this is on a testing branch in my personal spark form)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20725
**[Test build #87919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87919/testReport)** for PR 20725 at commit [`afcb0d5`](https://github.com/apache/spark/commit/afcb0d5608c17d2fc004a0b6d4af4573abca4e4b).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20725
This is working now, and BinaryType support is conditional on pyarrow 0.10.0 or higher being used. @HyukjinKwon @cloud-fan what are your thoughts on getting this in for Spark 2.4? I think it would be very useful to have since images in Spark use the BInaryType and it will be good to have when integrating Spark with DL frameworks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93854/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/20725
i've been having trouble getting this build to pass, and i'm testing a few other modules on top of yours. if i have problems, i'll revert the extraneous changes and build yours.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20725: [SPARK-23555][PYTHON] Add BinaryType support for ...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/20725#discussion_r210679722
--- Diff: python/pyspark/sql/tests.py ---
@@ -4331,13 +4354,22 @@ def test_createDataFrame_fallback_enabled(self):
self.assertEqual(df.collect(), [Row(a={u'a': 1})])
def test_createDataFrame_fallback_disabled(self):
+ from distutils.version import LooseVersion
import pandas as pd
+ import pyarrow as pa
with QuietTest(self.sc):
with self.assertRaisesRegexp(TypeError, 'Unsupported type'):
self.spark.createDataFrame(
pd.DataFrame([[{u'a': 1}]]), "a: map<string, int>")
+ # TODO: remove BinaryType check once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ with QuietTest(self.sc):
+ with self.assertRaisesRegexp(TypeError, 'Unsupported type.*BinaryType'):
+ self.spark.createDataFrame(
+ pd.DataFrame([[{'a': b'aaa'}]]), "a: binary")
--- End diff --
In general, Spark only accepts `bytearray` and pyarrow can accept `bytearray` or `bytes`. In pyarrow < 0.10.0 bytearrays weren't supported though, which is why I used `bytes` for the pandas pdf here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2236/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20725
@shaneknapp would you mind kicking off a test on one of the workers with python 3.5 and pyarrow 0.10.0 to verify this PR is passing under that env? Sorry for the special request, this should be the last thing to try to squeeze in before you upgrade jenkins.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20725
Awesome, thanks for running that @shaneknapp !! I'll merge this later today if no further comments.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20725: [SPARK-23555][PYTHON] Add BinaryType support for ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20725#discussion_r210489614
--- Diff: python/pyspark/sql/tests.py ---
@@ -4331,13 +4354,22 @@ def test_createDataFrame_fallback_enabled(self):
self.assertEqual(df.collect(), [Row(a={u'a': 1})])
def test_createDataFrame_fallback_disabled(self):
+ from distutils.version import LooseVersion
import pandas as pd
+ import pyarrow as pa
with QuietTest(self.sc):
with self.assertRaisesRegexp(TypeError, 'Unsupported type'):
self.spark.createDataFrame(
pd.DataFrame([[{u'a': 1}]]), "a: map<string, int>")
+ # TODO: remove BinaryType check once minimum pyarrow version is 0.10.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.10.0"):
+ with QuietTest(self.sc):
+ with self.assertRaisesRegexp(TypeError, 'Unsupported type.*BinaryType'):
+ self.spark.createDataFrame(
+ pd.DataFrame([[{'a': b'aaa'}]]), "a: binary")
--- End diff --
@BryanCutler, BTW, what kind of data can be binary? `bytearray` and `byte` (`str` in Python 2)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20725
**[Test build #87919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87919/testReport)** for PR 20725 at commit [`afcb0d5`](https://github.com/apache/spark/commit/afcb0d5608c17d2fc004a0b6d4af4573abca4e4b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20725
merged to master, thanks @shaneknapp and @HyukjinKwon !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20725
**[Test build #93854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93854/testReport)** for PR 20725 at commit [`45eea0c`](https://github.com/apache/spark/commit/45eea0c99f738cebd97bc643ad413a5f3957a045).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1248/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1540/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20725
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87919/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [SPARK-23555][PYTHON] Add BinaryType support for Arrow i...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/20725
still having problems getting this to pass:
```
[error] (sql-kafka-0-10/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] (core/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 14786 s, completed Aug 16, 2018 4:05:09 PM
```
i rebased upstream changes from the main spark repo on my fork and launched another build. in ~5 hours we'll know how it went. :\
if this fails tonite, i'll figure out a hacky way to test this tomorrow morning.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20725: [WIP][SPARK-23555][PYTHON] Add BinaryType support for Ar...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20725
This is a WIP as some issues need to be worked out on the Arrow side and need tests for pandas_udfs. Currently get the following error when converting from pandas:
```
File "/home/bryan/git/spark/python/pyspark/serializers.py", line 237, in create_array
return pa.Array.from_pandas(s, mask=mask, type=t)
File "array.pxi", line 335, in pyarrow.lib.Array.from_pandas
File "array.pxi", line 170, in pyarrow.lib.array
File "array.pxi", line 70, in pyarrow.lib._ndarray_to_array
File "error.pxi", line 85, in pyarrow.lib.check_status
ArrowNotImplementedError: No cast implemented from binary to binary
```
The corresponding JIRA is https://issues.apache.org/jira/browse/ARROW-2141
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org