You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2017/11/21 23:24:53 UTC
[arrow] branch master updated: ARROW-1268: [SITE][FOLLOWUP] Update
Spark Post to Reflect Conf Change
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 3fb1491 ARROW-1268: [SITE][FOLLOWUP] Update Spark Post to Reflect Conf Change
3fb1491 is described below
commit 3fb14914abfe89797f18a6cc5f62ec5cbe0ab2be
Author: Bryan Cutler <cu...@gmail.com>
AuthorDate: Tue Nov 21 18:24:49 2017 -0500
ARROW-1268: [SITE][FOLLOWUP] Update Spark Post to Reflect Conf Change
The Spark conf to enable arrow has changed from "spark.sql.execution.arrow.enable" to "spark.sql.execution.arrow.enabled"
Author: Bryan Cutler <cu...@gmail.com>
Closes #1344 from BryanCutler/site-Spark-blog-conf-update-ARROW-1268 and squashes the following commits:
5e784351 [Bryan Cutler] update spark conf to enable arrow
---
site/_posts/2017-07-26-spark-arrow.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/site/_posts/2017-07-26-spark-arrow.md b/site/_posts/2017-07-26-spark-arrow.md
index c4b16c0..211e5a4 100644
--- a/site/_posts/2017-07-26-spark-arrow.md
+++ b/site/_posts/2017-07-26-spark-arrow.md
@@ -57,7 +57,7 @@ the conversion to Arrow data can be done on the JVM and pushed back for the Spar
executors to perform in parallel, drastically reducing the load on the driver.
As of the merging of [SPARK-13534][5], the use of Arrow when calling `toPandas()`
-needs to be enabled by setting the SQLConf "spark.sql.execution.arrow.enable" to
+needs to be enabled by setting the SQLConf "spark.sql.execution.arrow.enabled" to
"true". Let's look at a simple usage example.
```
@@ -84,7 +84,7 @@ In [2]: %time pdf = df.toPandas()
CPU times: user 17.4 s, sys: 792 ms, total: 18.1 s
Wall time: 20.7 s
-In [3]: spark.conf.set("spark.sql.execution.arrow.enable", "true")
+In [3]: spark.conf.set("spark.sql.execution.arrow.enabled", "true")
In [4]: %time pdf = df.toPandas()
CPU times: user 40 ms, sys: 32 ms, total: 72 ms
@@ -118,7 +118,7 @@ It is planned to add pyarrow as a pyspark dependency so that
Currently, the controlling SQLConf is disabled by default. This can be enabled
programmatically as in the example above or by adding the line
-"spark.sql.execution.arrow.enable=true" to `SPARK_HOME/conf/spark-defaults.conf`.
+"spark.sql.execution.arrow.enabled=true" to `SPARK_HOME/conf/spark-defaults.conf`.
Also, not all Spark data types are currently supported and limited to primitive
types. Expanded type support is in the works and expected to also be in the Spark
--
To stop receiving notification emails like this one, please contact
['"commits@arrow.apache.org" <co...@arrow.apache.org>'].