You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/03/20 05:44:49 UTC
[GitHub] spark pull request #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf ut...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/20863
[SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in PySpark tests where possible
## What changes were proposed in this pull request?
This PR backports https://github.com/apache/spark/pull/20830 to reduce the diff against master and restore the default value back in PySpark tests.
https://github.com/apache/spark/commit/d6632d185e147fcbe6724545488ad80dce20277e added an useful util. This backport extracts and brings this util:
```python
@contextmanager
def sql_conf(self, pairs):
...
```
to allow configuration set/unset within a block:
```python
with self.sql_conf({"spark.blah.blah.blah", "blah"})
# test codes
```
This PR proposes to use this util where possible in PySpark tests.
Note that there look already few places affecting tests without restoring the original value back in unittest classes.
## How was this patch tested?
Likewise, manually tested via:
```
./run-tests --modules=pyspark-sql --python-executables=python2
./run-tests --modules=pyspark-sql --python-executables=python3
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark backport-20830
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20863.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20863
----
commit 4e8045cbddc39a5b8f3488b832a1ac092da68de9
Author: hyukjinkwon <gu...@...>
Date: 2018-03-20T04:25:37Z
[SPARK-23691][PYTHON] Use sql_conf util in PySpark tests where possible
https://github.com/apache/spark/commit/d6632d185e147fcbe6724545488ad80dce20277e added an useful util
```python
contextmanager
def sql_conf(self, pairs):
...
```
to allow configuration set/unset within a block:
```python
with self.sql_conf({"spark.blah.blah.blah", "blah"})
# test codes
```
This PR proposes to use this util where possible in PySpark tests.
Note that there look already few places affecting tests without restoring the original value back in unittest classes.
Manually tested via:
```
./run-tests --modules=pyspark-sql --python-executables=python2
./run-tests --modules=pyspark-sql --python-executables=python3
```
Author: hyukjinkwon <gu...@gmail.com>
Closes #20830 from HyukjinKwon/cleanup-sql-conf.
commit 3056e3c469209d72c97046f9668e30e0dbc5818d
Author: hyukjinkwon <gu...@...>
Date: 2018-03-20T05:27:26Z
Extracts and brings sql_conf util
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20863
**[Test build #88404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88404/testReport)** for PR 20863 at commit [`3056e3c`](https://github.com/apache/spark/commit/3056e3c469209d72c97046f9668e30e0dbc5818d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20863
cc @ueshin and @BryanCutler
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf ut...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20863#discussion_r175660234
--- Diff: python/pyspark/sql/tests.py ---
@@ -2409,17 +2432,13 @@ def test_join_without_on(self):
df1 = self.spark.range(1).toDF("a")
df2 = self.spark.range(1).toDF("b")
- try:
--- End diff --
Other diff are basically the same.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf ut...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20863#discussion_r175660221
--- Diff: python/pyspark/sql/tests.py ---
@@ -201,6 +202,28 @@ def assertPandasEqual(self, expected, result):
"\n\nResult:\n%s\n%s" % (result, result.dtypes))
self.assertTrue(expected.equals(result), msg=msg)
+ @contextmanager
--- End diff --
This was extracted alone from https://github.com/apache/spark/commit/d6632d185e147fcbe6724545488ad80dce20277e
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20863
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1631/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf ut...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/spark/pull/20863
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20863
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88404/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20863
**[Test build #88404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88404/testReport)** for PR 20863 at commit [`3056e3c`](https://github.com/apache/spark/commit/3056e3c469209d72c97046f9668e30e0dbc5818d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20863
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20863
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20863: [SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in P...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20863
Merged to branch-2.3.
Thank you for reviewing this @ueshin.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org