You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by andrewor14 <gi...@git.apache.org> on 2016/04/29 21:54:53 UTC
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/12784
[SPARK-14988][PYTHON] SparkSession API follow-ups
## What changes were proposed in this pull request?
Addresses comments in #12765.
## How was this patch tested?
Python tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark python-followup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12784.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12784
----
commit 43f8ea16ede81cfbf40b1c4514bd31643f4ed14a
Author: Andrew Or <an...@databricks.com>
Date: 2016-04-29T16:43:12Z
registerDataFrameAsTable -> registerTable
commit 7b7d285e214d6f2cf6075558e7c98346b0449262
Author: Andrew Or <an...@databricks.com>
Date: 2016-04-29T17:10:43Z
getOption -> get with default
commit 4aeda61a56f7ae51d5594cbe94115bd6f35071c2
Author: Andrew Or <an...@databricks.com>
Date: 2016-04-29T17:48:26Z
Move conf tests
commit 309fd35fc69423e875c5de15d5249ae0af859c19
Author: Andrew Or <an...@databricks.com>
Date: 2016-04-29T19:12:53Z
Move catalog tests
commit ee606dd9425cb5ec28273477dd4b560b4cb6bd3e
Author: Andrew Or <an...@databricks.com>
Date: 2016-04-29T19:16:57Z
Fix style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215906453
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57368/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215881529
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215895402
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215908390
Merging into master. Thanks @davies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215892558
**[Test build #57368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57368/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215904773
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57365/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215881364
**[Test build #57355 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57355/consoleFull)** for PR 12784 at commit [`ee606dd`](https://github.com/apache/spark/commit/ee606dd9425cb5ec28273477dd4b560b4cb6bd3e).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215906482
**[Test build #2940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2940/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215895275
**[Test build #57360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57360/consoleFull)** for PR 12784 at commit [`a8b0732`](https://github.com/apache/spark/commit/a8b0732392599e3af86137b0d207aa16eb27f5da).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12784#discussion_r61645360
--- Diff: python/pyspark/sql/conf.py ---
@@ -33,62 +33,24 @@ def __init__(self, jconf):
@ignore_unicode_prefix
@since(2.0)
def set(self, key, value):
- """Sets the given Spark runtime configuration property.
-
- >>> spark.conf.set("garble", "marble")
- >>> spark.getConf("garble")
- u'marble'
- """
+ """Sets the given Spark runtime configuration property."""
self._jconf.set(key, value)
@ignore_unicode_prefix
@since(2.0)
- def get(self, key):
+ def get(self, key, default=None):
"""Returns the value of Spark runtime configuration property for the given key,
assuming it is set.
-
- >>> spark.setConf("bogo", "sipeo")
- >>> spark.conf.get("bogo")
- u'sipeo'
- >>> spark.conf.get("definitely.not.set") # doctest: +IGNORE_EXCEPTION_DETAIL
- Traceback (most recent call last):
- ...
- Py4JJavaError: ...
"""
- return self._jconf.get(key)
-
- @ignore_unicode_prefix
- @since(2.0)
- def getOption(self, key):
- """Returns the value of Spark runtime configuration property for the given key,
- or None if it is not set.
-
- >>> spark.setConf("bogo", "sipeo")
- >>> spark.conf.getOption("bogo")
- u'sipeo'
- >>> spark.conf.getOption("definitely.not.set") is None
- True
- """
- iter = self._jconf.getOption(key).iterator()
- if iter.hasNext():
- return iter.next()
+ if default is None:
+ return self._jconf.get(key)
else:
- return None
+ return self._jconf.get(key, default)
--- End diff --
Could you check the type for key and default, or you will get an weird py4j error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215905867
**[Test build #2942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2942/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215892632
**[Test build #2942 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2942/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215904771
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215906452
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215906216
**[Test build #2941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2941/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215881531
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57355/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215862253
@davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215863298
**[Test build #57355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57355/consoleFull)** for PR 12784 at commit [`ee606dd`](https://github.com/apache/spark/commit/ee606dd9425cb5ec28273477dd4b560b4cb6bd3e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215904610
**[Test build #57365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57365/consoleFull)** for PR 12784 at commit [`768fd81`](https://github.com/apache/spark/commit/768fd816a3c7b0c49eb5eaf43cb322d17e94d9a2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215892605
**[Test build #2941 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2941/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215906303
**[Test build #57368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57368/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215881001
**[Test build #57360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57360/consoleFull)** for PR 12784 at commit [`a8b0732`](https://github.com/apache/spark/commit/a8b0732392599e3af86137b0d207aa16eb27f5da).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12784#discussion_r61645639
--- Diff: python/pyspark/sql/tests.py ---
@@ -1394,6 +1395,200 @@ def test_toDF_with_schema_string(self):
self.assertEqual(df.schema.simpleString(), "struct<value:int>")
self.assertEqual(df.collect(), [Row(key=i) for i in range(100)])
+ def test_conf(self):
+ spark = self.sparkSession
+ spark.setConf("bogo", "sipeo")
+ self.assertEqual(self.sparkSession.conf.get("bogo"), "sipeo")
+ spark.setConf("bogo", "ta")
+ self.assertEqual(spark.conf.get("bogo"), "ta")
+ self.assertEqual(spark.conf.get("bogo", "not.read"), "ta")
+ self.assertEqual(spark.conf.get("not.set", "ta"), "ta")
+ self.assertRaisesRegexp(Exception, "not.set", lambda: spark.conf.get("not.set"))
+ spark.conf.unset("bogo")
+ self.assertEqual(spark.conf.get("bogo", "colombia"), "colombia")
+
+ def test_current_database(self):
+ spark = self.sparkSession
+ spark.catalog._reset()
+ self.assertEquals(spark.catalog.currentDatabase(), "default")
+ spark.sql("CREATE DATABASE some_db")
+ spark.catalog.setCurrentDatabase("some_db")
+ self.assertEquals(spark.catalog.currentDatabase(), "some_db")
+ self.assertRaisesRegexp(
+ AnalysisException,
+ "does_not_exist",
+ lambda: spark.catalog.setCurrentDatabase("does_not_exist"))
+
+ def test_list_databases(self):
+ spark = self.sparkSession
+ spark.catalog._reset()
+ databases = [db.name for db in spark.catalog.listDatabases()]
+ self.assertEquals(databases, ["default"])
+ spark.sql("CREATE DATABASE some_db")
+ databases = [db.name for db in spark.catalog.listDatabases()]
+ self.assertEquals(sorted(databases), ["default", "some_db"])
+
+ def test_list_tables(self):
+ from catalog import Table
--- End diff --
It's better to use full path
from pyspark.sql.catalog import Table
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215895404
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57360/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215892579
**[Test build #2940 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2940/consoleFull)** for PR 12784 at commit [`059bfdd`](https://github.com/apache/spark/commit/059bfdd73d10941bc1d314735d78bafc99149d9b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12784#issuecomment-215890233
**[Test build #57365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57365/consoleFull)** for PR 12784 at commit [`768fd81`](https://github.com/apache/spark/commit/768fd816a3c7b0c49eb5eaf43cb322d17e94d9a2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14988][PYTHON] SparkSession API follow-...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12784
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org