You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2018/10/29 06:01:39 UTC
spark git commit: [SPARK-25179][PYTHON][DOCS] Document BinaryType
support in Arrow conversion
Repository: spark
Updated Branches:
refs/heads/master 4e990d9dd -> fbaf15050
[SPARK-25179][PYTHON][DOCS] Document BinaryType support in Arrow conversion
## What changes were proposed in this pull request?
This PR targets to document binary type in "Apache Arrow in Spark".
## How was this patch tested?
Manually built the documentation and checked.
Closes #22871 from HyukjinKwon/SPARK-25179.
Authored-by: hyukjinkwon <gu...@apache.org>
Signed-off-by: gatorsmile <ga...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fbaf1505
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fbaf1505
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fbaf1505
Branch: refs/heads/master
Commit: fbaf150507a289ec0ac02fdbf4009c42cd9bc164
Parents: 4e990d9
Author: hyukjinkwon <gu...@apache.org>
Authored: Sun Oct 28 23:01:35 2018 -0700
Committer: gatorsmile <ga...@gmail.com>
Committed: Sun Oct 28 23:01:35 2018 -0700
----------------------------------------------------------------------
docs/sql-pyspark-pandas-with-arrow.md | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/fbaf1505/docs/sql-pyspark-pandas-with-arrow.md
----------------------------------------------------------------------
diff --git a/docs/sql-pyspark-pandas-with-arrow.md b/docs/sql-pyspark-pandas-with-arrow.md
index e8e9f55..d04b955 100644
--- a/docs/sql-pyspark-pandas-with-arrow.md
+++ b/docs/sql-pyspark-pandas-with-arrow.md
@@ -127,8 +127,9 @@ For detailed usage, please see [`pyspark.sql.functions.pandas_udf`](api/python/p
### Supported SQL Types
-Currently, all Spark SQL data types are supported by Arrow-based conversion except `BinaryType`, `MapType`,
-`ArrayType` of `TimestampType`, and nested `StructType`.
+Currently, all Spark SQL data types are supported by Arrow-based conversion except `MapType`,
+`ArrayType` of `TimestampType`, and nested `StructType`. `BinaryType` is supported only when
+installed PyArrow is equal to or higher then 0.10.0.
### Setting Arrow Batch Size
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org