You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/12 08:03:40 UTC
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37490: [SPARK-40051][PYTHON][SQL][DOCS] Make pyspark.sql.catalog examples self-contained
zhengruifeng commented on code in PR #37490:
URL: https://github.com/apache/spark/pull/37490#discussion_r944206889
##########
python/pyspark/sql/catalog.py:
##########
@@ -184,38 +234,65 @@ def databaseExists(self, dbName: str) -> bool:
Parameters
----------
dbName : str
- name of the database to check existence
+ name of the database to check existence
+
+ .. versionchanged:: 3.4.0
+ Allow ``dbName`` to be qualified with catalog name.
Returns
-------
bool
Indicating whether the database exists
- .. versionchanged:: 3.4
- Allowed ``dbName`` to be qualified with catalog name.
-
Examples
--------
+ Check if 'test_new_database' database exists
+
>>> spark.catalog.databaseExists("test_new_database")
False
- >>> df = spark.sql("CREATE DATABASE test_new_database")
+ >>> _ = spark.sql("CREATE DATABASE test_new_database")
>>> spark.catalog.databaseExists("test_new_database")
True
+
+ Using the fully qualified name with the catalog name.
+
>>> spark.catalog.databaseExists("spark_catalog.test_new_database")
True
- >>> df = spark.sql("DROP DATABASE test_new_database")
+ >>> _ = spark.sql("DROP DATABASE test_new_database")
"""
return self._jcatalog.databaseExists(dbName)
@since(2.0)
Review Comment:
does this annotation also need to be removed?
##########
python/pyspark/sql/catalog.py:
##########
@@ -674,59 +875,267 @@ def registerFunction(
warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
return self._sparkSession.udf.register(name, f, returnType)
- @since(2.0)
def isCached(self, tableName: str) -> bool:
- """Returns true if the table is currently cached in-memory.
+ """
+ Returns true if the table is currently cached in-memory.
+
+ .. versionadded:: 2.0.0
+
+ Parameters
+ ----------
+ tableName : str
+ name of the table to get.
+
+ .. versionchanged:: 3.4.0
+ Allow ``tableName`` to be qualified with catalog name.
+
+ Returns
+ -------
+ bool
+
+ Examples
+ --------
+ >>> _ = spark.sql("DROP TABLE IF EXISTS tbl1")
+ >>> _ = spark.sql("CREATE TABLE tbl1 (name STRING, age INT) USING parquet")
+ >>> spark.catalog.cacheTable("tbl1")
+ >>> spark.catalog.isCached("tbl1")
+ True
+
+ Throw an analysis exception when the table does not exists.
- .. versionchanged:: 3.4
- Allowed ``tableName`` to be qualified with catalog name.
+ >>> spark.catalog.isCached("not_existing_table")
+ Traceback (most recent call last):
+ ...
+ pyspark.sql.utils.AnalysisException: ...
Review Comment:
can this be simplified to `AnalysisException`?
##########
python/pyspark/sql/catalog.py:
##########
@@ -404,26 +529,42 @@ def getFunction(self, functionName: str) -> Function:
def listColumns(self, tableName: str, dbName: Optional[str] = None) -> List[Column]:
"""Returns a list of columns for the given table/view in the specified database.
- If no database is specified, the current database is used.
-
.. versionadded:: 2.0.0
Parameters
----------
tableName : str
- name of the table to check existence
+ name of the table to list columns.
+
+ .. versionchanged:: 3.4.0
+ Allow ``tableName`` to be qualified with catalog name when ``dbName`` is None.
+
dbName : str, optional
- name of the database to check table existence in.
Review Comment:
dumb question: it is possbile to make `lint-python` also check the style in docstring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org