You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/28 22:28:15 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
xinrong-databricks opened a new pull request #34133:
URL: https://github.com/apache/spark/pull/34133
### What changes were proposed in this pull request?
Inline type hints for python/pyspark/sql/catalog.py.
### Why are the changes needed?
Currently, a type hint stub file hints for python/pyspark/sql/catalog.pyi is used. We may leverage static type check by inlining type hints.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929684748
**[Test build #143687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143687/testReport)** for PR 34133 at commit [`87a5653`](https://github.com/apache/spark/commit/87a5653a6088fabf5f0d3d00e551db3bf09a6332).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933988959
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718942974
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Good catch! Would you mind if fixing that in https://github.com/apache/spark/pull/34147? There are two more APIs to be fixed similarly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930650029
> Aren't we concerned about impact on backporting any more?
Yeah, this is a good point ... Probably yeah we will stop backporting it I guess ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718965674
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
I think it's fine to remove `return` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929716582
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r721789621
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
Review comment:
Sounds good, updated https://github.com/apache/spark/pull/34133/commits/1403d2650e093b25130da8c733309e6cb7b9cb7f!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929690277
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933952620
**[Test build #143824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143824/testReport)** for PR 34133 at commit [`1403d26`](https://github.com/apache/spark/commit/1403d2650e093b25130da8c733309e6cb7b9cb7f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930487028
Thanks for letting me know @HyukjinKwon. Aren't we concerned about impact on backporting any more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929673031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929692224
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143687/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930625580
**[Test build #143734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143734/testReport)** for PR 34133 at commit [`a619ea8`](https://github.com/apache/spark/commit/a619ea8c7cd054fb76ea3169ed1c903465780ce1).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718963596
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
Review comment:
Sounds good! Actually I agree and prefer it.
If we would, we should do it in a separate PR to make it clear whether the change works as expected or not.
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
I think it's fine to remove `return` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929722104
cc @zero323 FYI. We're now trying to inline type hints for module (or file by file) in batch. Seems like type hints are very useful and stable then I thought :-). so I think we're pretty safe to inline them now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930667280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933961276
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718833451
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
Review comment:
We should probably have these two (as well as `Database` and `Table`) annotated here. If we're going to in-line all annotations, it will require choosing a convention. Personally, I prefer 3.6 syntax:
```python
class Coulumn(NamedTuple):
name: str
description: str
dataType: str
nullable: bool
isPartition: bool
isBucket: bool
```
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Can we fix this as well? We should have no `return` if function is not expected to return anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933988959
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34133:
URL: https://github.com/apache/spark/pull/34133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929722104
cc @zero323 FYI. We're now trying to inline type hints for module (or file by file) in batch. Seems like type hints are very useful and stabler then I thought :-). so I think we're pretty safe to inline them now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930625797
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143734/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r721789621
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
Review comment:
Sounds good, updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718833967
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Can we fix this as well? We should have no `return` if function is not expected to return anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718963596
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
Review comment:
Sounds good! Actually I agree and prefer it.
If we would, we should do it in a separate PR to make it clear whether the change works as expected or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718950573
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Is it fine to remove the `return` in this PR since there are no user-facing changes introduced? CC @HyukjinKwon @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718942974
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Good catch! Updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930667280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929692224
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143687/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933940722
**[Test build #143824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143824/testReport)** for PR 34133 at commit [`1403d26`](https://github.com/apache/spark/commit/1403d2650e093b25130da8c733309e6cb7b9cb7f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718833451
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
Review comment:
We should probably have these two (as well as `Database` and `Table`) annotated here. If we're going to in-line all annotations, it will require choosing a convention. Personally, I prefer 3.6 syntax:
```python
class Coulumn(NamedTuple):
name: str
description: str
dataType: str
nullable: bool
isPartition: bool
isBucket: bool
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718942974
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Good catch! Would you mind if fixing that in https://github.com/apache/spark/pull/34147? There are two more APIs to be fixed similarly.
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Good catch! Would you mind if fixing that in https://github.com/apache/spark/pull/34147? There are two more APIs to be fixed similarly. In this way, we may also keep user-facing changes separate from this PR.
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Good catch! Updated it.
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Would you help confirm if we may remove the `return` here since there are no user-facing changes introduced? CC @HyukjinKwon @ueshin
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Would you help confirm if we may remove the `return` in this PR since there are no user-facing changes introduced? CC @HyukjinKwon @ueshin
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Is it fine to remove the `return` in this PR since there are no user-facing changes introduced? CC @HyukjinKwon @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930622644
**[Test build #143736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143736/testReport)** for PR 34133 at commit [`59ab500`](https://github.com/apache/spark/commit/59ab500083ca191a016063f747b8ede7079ebe1a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933965297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143824/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933980463
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933965297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143824/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929706741
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929673031
**[Test build #143687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143687/testReport)** for PR 34133 at commit [`87a5653`](https://github.com/apache/spark/commit/87a5653a6088fabf5f0d3d00e551db3bf09a6332).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929716582
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930663070
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48246/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929673031
**[Test build #143687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143687/testReport)** for PR 34133 at commit [`87a5653`](https://github.com/apache/spark/commit/87a5653a6088fabf5f0d3d00e551db3bf09a6332).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930625797
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143734/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718942974
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Good catch! Would you mind if fixing that in https://github.com/apache/spark/pull/34147? There are two more APIs to be fixed similarly. In this way, we may also keep user-facing changes separate from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929692224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933989041
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929673031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r720561541
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
Review comment:
nit: shall we move these to between after the other imports and the definition of `Database`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718950573
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Would you help confirm if we may remove the `return` here since there are no user-facing changes introduced? CC @HyukjinKwon @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930487028
Thanks for letting me know @HyukjinKwon. Aren't we concerned about impact on backporting any more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930617727
**[Test build #143734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143734/testReport)** for PR 34133 at commit [`a619ea8`](https://github.com/apache/spark/commit/a619ea8c7cd054fb76ea3169ed1c903465780ce1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930641172
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143736/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930617727
**[Test build #143734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143734/testReport)** for PR 34133 at commit [`a619ea8`](https://github.com/apache/spark/commit/a619ea8c7cd054fb76ea3169ed1c903465780ce1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718950573
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
+if TYPE_CHECKING:
+ from pyspark.sql._typing import UserDefinedFunctionLike
+ from pyspark.sql.types import DataType
+
class Catalog(object):
"""User-facing catalog API, accessible through `SparkSession.catalog`.
This is a thin wrapper around its Scala implementation org.apache.spark.sql.catalog.Catalog.
"""
- def __init__(self, sparkSession):
+ def __init__(self, sparkSession: SparkSession) -> None:
"""Create a new Catalog that wraps the underlying JVM object."""
self._sparkSession = sparkSession
- self._jsparkSession = sparkSession._jsparkSession
- self._jcatalog = sparkSession._jsparkSession.catalog()
+ self._jsparkSession = sparkSession._jsparkSession # type: ignore[attr-defined]
+ self._jcatalog = sparkSession._jsparkSession.catalog() # type: ignore[attr-defined]
@since(2.0)
- def currentDatabase(self):
+ def currentDatabase(self) -> str:
"""Returns the current default database in this session."""
return self._jcatalog.currentDatabase()
@since(2.0)
- def setCurrentDatabase(self, dbName):
+ def setCurrentDatabase(self, dbName: str) -> None:
"""Sets the current default database in this session."""
return self._jcatalog.setCurrentDatabase(dbName)
Review comment:
Would you help confirm if we may remove the `return` in this PR since there are no user-facing changes introduced? CC @HyukjinKwon @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929692224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34133:
URL: https://github.com/apache/spark/pull/34133#discussion_r718997879
##########
File path: python/pyspark/sql/catalog.py
##########
@@ -29,31 +31,35 @@
Column = namedtuple("Column", "name description dataType nullable isPartition isBucket")
Function = namedtuple("Function", "name description className isTemporary")
Review comment:
Great! I create https://issues.apache.org/jira/browse/SPARK-36897 to track that. Thanks for the input!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930663269
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48247/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-929722104
cc @zero323 FYI. We're now trying to inline type hints for module (or file by file) in batch. Seems like type hints are very useful and stabler then I thought :-). so I think we're pretty safe to inline them now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-930643442
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48247/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34133: [SPARK-36881][PYTHON] Inline type hints for python/pyspark/sql/catalog.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34133:
URL: https://github.com/apache/spark/pull/34133#issuecomment-933940722
**[Test build #143824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143824/testReport)** for PR 34133 at commit [`1403d26`](https://github.com/apache/spark/commit/1403d2650e093b25130da8c733309e6cb7b9cb7f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org