You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ueshin <gi...@git.apache.org> on 2017/10/27 08:52:43 UTC
[GitHub] spark pull request #19587: [SPARK-22370][SQL][PYSPARK] Config values should ...
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/19587
[SPARK-22370][SQL][PYSPARK] Config values should be captured in Driver.
## What changes were proposed in this pull request?
We should capture config values of `SQLConf` in Driver.
## How was this patch tested?
Added a test and existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-22370
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19587.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19587
----
commit 091887255ff2caa9b3e6691321407b86977bf513
Author: Takuya UESHIN <ue...@databricks.com>
Date: 2017-10-27T08:38:53Z
Config values should be captured in Driver.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19587
LGTM, good catch!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19587
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19587: [SPARK-22370][SQL][PYSPARK] Config values should ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19587
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83118/testReport)** for PR 19587 at commit [`0918872`](https://github.com/apache/spark/commit/091887255ff2caa9b3e6691321407b86977bf513).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83118/testReport)** for PR 19587 at commit [`0918872`](https://github.com/apache/spark/commit/091887255ff2caa9b3e6691321407b86977bf513).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19587
LGTM too
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19587
Could you add a comment in `abstract class QueryPlan`'s conf? Hopefully, the others will read the comment before using it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83160/testReport)** for PR 19587 at commit [`624736e`](https://github.com/apache/spark/commit/624736ea31151f978dbdc533336d81dda7f8d192).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83118/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83154/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19587: [SPARK-22370][SQL][PYSPARK] Config values should ...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19587#discussion_r147463026
--- Diff: python/pyspark/sql/tests.py ---
@@ -3476,6 +3476,26 @@ def gen_timestamps(id):
expected = spark_ts_t.fromInternal(spark_ts_t.toInternal(ts_tz))
self.assertEquals(expected, ts)
+ def test_vectorized_udf_check_config(self):
+ from pyspark.sql.functions import pandas_udf, col
+ orig_value = self.spark.conf.get("spark.sql.execution.arrow.maxRecordsPerBatch", None)
+ self.spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", 3)
+ try:
+ df = self.spark.range(10, numPartitions=1)
+
+ @pandas_udf(returnType=LongType())
+ def checkRecordsPerBatch(x):
--- End diff --
Thanks! I'll update it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by icexelloss <gi...@git.apache.org>.
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/19587
LGTM. +1
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19587
LGTM, merging to master!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83160/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19587
cc @BryanCutler @icexelloss @cloud-fan @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19587: [SPARK-22370][SQL][PYSPARK] Config values should ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19587#discussion_r147411635
--- Diff: python/pyspark/sql/tests.py ---
@@ -3476,6 +3476,26 @@ def gen_timestamps(id):
expected = spark_ts_t.fromInternal(spark_ts_t.toInternal(ts_tz))
self.assertEquals(expected, ts)
+ def test_vectorized_udf_check_config(self):
+ from pyspark.sql.functions import pandas_udf, col
+ orig_value = self.spark.conf.get("spark.sql.execution.arrow.maxRecordsPerBatch", None)
+ self.spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", 3)
+ try:
+ df = self.spark.range(10, numPartitions=1)
+
+ @pandas_udf(returnType=LongType())
+ def checkRecordsPerBatch(x):
--- End diff --
nit: `checkRecordsPerBatch` -> `check_records_per_batch`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83130/testReport)** for PR 19587 at commit [`b47f55b`](https://github.com/apache/spark/commit/b47f55b8e2eb0de99ddb94cd1b96423f0ddcf1a4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83154/testReport)** for PR 19587 at commit [`ea20b03`](https://github.com/apache/spark/commit/ea20b039f07424eb574679cd53e31b704e50abf2).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83160/testReport)** for PR 19587 at commit [`624736e`](https://github.com/apache/spark/commit/624736ea31151f978dbdc533336d81dda7f8d192).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19587
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83130/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83130/testReport)** for PR 19587 at commit [`b47f55b`](https://github.com/apache/spark/commit/b47f55b8e2eb0de99ddb94cd1b96423f0ddcf1a4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19587: [SPARK-22370][SQL][PYSPARK] Config values should be capt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19587
**[Test build #83154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83154/testReport)** for PR 19587 at commit [`ea20b03`](https://github.com/apache/spark/commit/ea20b039f07424eb574679cd53e31b704e50abf2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org