You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dbtsai (via GitHub)" <gi...@apache.org> on 2026/04/27 22:05:30 UTC
[PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
dbtsai opened a new pull request, #55574:
URL: https://github.com/apache/spark/pull/55574
Implements a Connect-compatible SQLContext in pyspark.sql.connect.context that wraps a Connect SparkSession instead of requiring a SparkContext. All SparkSession-delegate methods (sql, table, range, createDataFrame, conf, udf, udtf, read, readStream, streams, catalog ops) are wired up. JVM-only APIs (registerJavaFunction, HiveContext) raise PySparkNotImplementedError. Adds 20 unit tests.
Co-authored-by: Isaac
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3331724741
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,459 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.2.0
Review Comment:
`versionadded` should be `4.3.0`, not `4.2.0`: `4.2.0` is in code freeze and this PR isn't a backport candidate, so it'll land in the next open feature release. This applies to every `.. versionadded:: 4.2.0` in this file (~22 occurrences), not just this one.
```suggestion
.. versionadded:: 4.3.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3321011002
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
Review Comment:
Fixed.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
Review Comment:
Fixed.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
Review Comment:
Fixed.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ rows = [
+ (t.namespace[0] if t.namespace else "", t.name, t.isTemporary)
+ for t in self.sparkSession.catalog.listTables(dbName)
+ ]
+ return self.sparkSession.createDataFrame(rows, schema)
+
+ def tableNames(self, dbName: Optional[str] = None) -> List[str]:
+ """Returns a list of names of tables in the database ``dbName``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str
+ name of the database to use. Default to the current database.
+
+ Returns
+ -------
+ list
+ list of table names, in string
Review Comment:
Fixed.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ rows = [
+ (t.namespace[0] if t.namespace else "", t.name, t.isTemporary)
+ for t in self.sparkSession.catalog.listTables(dbName)
+ ]
+ return self.sparkSession.createDataFrame(rows, schema)
+
+ def tableNames(self, dbName: Optional[str] = None) -> List[str]:
+ """Returns a list of names of tables in the database ``dbName``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str
+ name of the database to use. Default to the current database.
+
+ Returns
+ -------
+ list
+ list of table names, in string
+ """
+ return [t.name for t in self.sparkSession.catalog.listTables(dbName)]
+
+ def cacheTable(self, tableName: str) -> None:
+ """Caches the specified table in-memory.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.cacheTable(tableName)
+
+ def uncacheTable(self, tableName: str) -> None:
+ """Removes the specified table from the in-memory cache.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.uncacheTable(tableName)
+
+ def clearCache(self) -> None:
+ """Removes all cached tables from the in-memory cache.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.clearCache()
+
+ @property
+ def read(self) -> DataFrameReader:
+ """Returns a :class:`DataFrameReader` that can be used to read data
+ in as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrameReader`
+ """
+ return self.sparkSession.read
+
+ @property
+ def readStream(self) -> DataStreamReader:
+ """Returns a :class:`DataStreamReader` that can be used to read data streams
+ as a streaming :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Notes
+ -----
+ This API is evolving.
+
+ Returns
+ -------
+ :class:`DataStreamReader`
+ """
+ return self.sparkSession.readStream
+
+ @property
+ def streams(self) -> StreamingQueryManager:
+ """Returns a :class:`StreamingQueryManager` that allows managing all the
+ :class:`~pyspark.sql.streaming.StreamingQuery` StreamingQueries active on
Review Comment:
Fixed.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ rows = [
Review Comment:
Added a two-line comment explaining the choice: `SHOW TABLES` returns `database` vs `namespace` depending on the active catalog, so `catalog.listTables()` is used to guarantee the column names always match the classic implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306194750
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,37 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect.context import SQLContext as ConnectSQLContext
+
+ session = SparkSession._getActiveSessionOrCreate()
+ return ConnectSQLContext._get_or_create_from_session( # type: ignore[return-value]
+ session # type: ignore[arg-type]
Review Comment:
Fixed in f5d18f65dca. Used `cast()` (already imported in this file) in both places: `cast(ConnectSparkSession, session)` for the argument and `cast("SQLContext", ...)` for the return value. Also added the inline `ConnectSparkSession` import alongside the existing `ConnectSQLContext` import to make the cast explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3327358845
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,457 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
Review Comment:
Fixed in e33dd662623 — removed `__all__` entirely and replaced it with a comment marking the module as an internal implementation detail.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,457 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.0.0
Review Comment:
Fixed in e33dd662623 — changed all 22 occurrences of `versionadded:: 4.0.0` to `versionadded:: 5.0.0` in `connect/context.py`.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,44 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
Review Comment:
Fixed in e33dd662623 — added `test_getOrCreate_emits_deprecation_and_returns_connect_context` and `test_hive_context_getOrCreate_raises` to `test_connect_context.py`, both using `unittest.mock.patch('pyspark.util.is_remote_only', return_value=True)` to exercise the dispatch path.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,44 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect import context as _connect_context
+ from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
+
+ session = SparkSession._getActiveSessionOrCreate()
+ # Route to the Connect counterpart so subclasses (e.g. HiveContext) are handled
+ # correctly: ConnectHiveContext._from_session raises PySparkNotImplementedError.
Review Comment:
Fixed in e33dd662623 — applied the suggestion exactly.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -27,6 +27,7 @@
from pyspark.sql import Row, SparkSession
from pyspark.sql.types import StructType, StringType, StructField
from pyspark.testing.sqlutils import ReusedSQLTestCase
+from pyspark.sql.tests.test_sql_context import SQLContextTestsMixin # noqa: F401
Review Comment:
Fixed in e33dd662623 — removed the dead import.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306193531
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -171,6 +171,109 @@ def range_frame_match():
reload(window)
+class SQLContextTestsMixin:
+ """Tests for SQLContext that run in both classic and Connect modes.
+
+ Subclasses must implement :meth:`_make_ctx` to return an :class:`SQLContext`
+ appropriate for their mode, and must expose ``self.spark`` as a
+ :class:`SparkSession`.
+ """
+
+ spark: SparkSession
+
+ def _make_ctx(self) -> SQLContext:
+ raise NotImplementedError
+
+ def setUp(self) -> None:
+ SQLContext._instantiatedContext = None
Review Comment:
Fixed in f5d18f65dca. The mixin's `setUp`/`tearDown` correctly resets `pyspark.sql.context.SQLContext._instantiatedContext` (the classic class, imported via `from pyspark import SQLContext`). However, the Connect subclass `SQLContextParityTests` creates `pyspark.sql.connect.context.SQLContext` instances, whose own `_instantiatedContext` was never reset between tests. Added explicit `setUp`/`tearDown` overrides in `SQLContextParityTests` that call `super()` and then also reset the Connect class's `_instantiatedContext`.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ if dbName is None:
+ return self.sparkSession.sql("SHOW TABLES")
+ return self.sparkSession.sql(f"SHOW TABLES IN {dbName}")
Review Comment:
Fixed in f5d18f65dca. Switched from `SHOW TABLES` (whose column names vary — `namespace` vs `database` depending on the catalog) to `catalog.listTables()`, building the result explicitly with a fixed `StructType([namespace: String, tableName: String, isTemporary: Boolean])` schema. This matches the classic implementation which always returns `(namespace, tableName, isTemporary)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3309379057
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
Review Comment:
**Finding #1 — design / top-level question: what is the public `pyspark.sql.connect.context.SQLContext` for?**
The classic-`getOrCreate` dispatch already covers the only transparent-compat case: legacy code that calls `SQLContext.getOrCreate()` in a remote-only install keeps working unchanged. Every other path requires the user to edit their imports — and once they're editing, migrating the call site to `SparkSession` is a strictly smaller, less-deprecated diff than switching to `from pyspark.sql.connect.context import SQLContext` (the deprecation recommendation since 2.0 already points there). I can't see a user journey where the public Connect-side class is the right answer over `SparkSession`.
Two cleaner shapes worth considering:
(a) **Make it implementation-private** — rename `_SQLContext`, or inline the wrapper into the `is_remote_only()` branch of classic `getOrCreate`. Drop the per-method entries in `legacy.rst`. The contract becomes: legacy `getOrCreate` keeps working; everyone else writes `SparkSession`.
(b) **Keep it public but trim** to a one-paragraph stub in `legacy.rst` that points to `SparkSession`, instead of advertising the full method surface.
If (a): the signature-divergence and `getOrCreate`/`_get_or_create_from_session` duplication concerns in Finding #4 also disappear, since there are no public callers — we keep `_get_or_create_from_session` and delete the public `getOrCreate`.
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,39 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect.context import SQLContext as ConnectSQLContext
Review Comment:
**Finding #2 — `HiveContext` bypass through classic dispatch.**
This `is_remote_only()` branch hardcodes `ConnectSQLContext` regardless of `cls`. A remote-only user calling `from pyspark.sql import HiveContext; HiveContext.getOrCreate()` reaches here with `cls=HiveContext`, gets handed back a plain `ConnectSQLContext`, and never sees the `PySparkNotImplementedError` they should.
c967950ec9e closed the *direct*-Connect bypass via `_from_session`, but the classic dispatch bypasses `_from_session` entirely because it routes to `ConnectSQLContext`, not `ConnectHiveContext`.
Two fixes:
- (a) route based on `cls` — `if cls is HiveContext: from pyspark.sql.connect.context import HiveContext as ConnectHiveContext; ...`; or
- (b) override `HiveContext.getOrCreate` in the classic file to raise in remote-only mode before delegating.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
Review Comment:
**Finding #3 — `cloneSession()` is not equivalent to `newSession()`.**
Classic `SparkSession.newSession()` returns a session with **separate** SQLConf, temp views, and UDFs and a shared cache — i.e. a fresh-empty session (`pyspark/sql/session.py:717-735`).
Connect `cloneSession()` does the opposite: it **copies** the current session's conf, temp views, registered functions, and catalog state into an independent server session (`pyspark/sql/connect/session.py:1282-1314`).
The PR description and the docstring at line 124-126 both promise classic "separate SQLConf / temp views / UDFs" semantics, but users get the cloned state. Either update the docstring to describe the cloneSession behavior accurately, or reach for a server-side construct that gives fresh state.
Note also that `cloneSession` is documented as a "developer API" (`pyspark/sql/connect/session.py:1306`) — coupling a user-facing method to it makes us fragile to changes there.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
Review Comment:
Grammar — missing article:
```suggestion
"""Returns a new SQLContext as a new session, that has separate SQLConf,
```
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
Review Comment:
**Finding #4 — signature divergence + duplication (subsumed if Finding #1 takes path (a)).**
(a) Classic `getOrCreate(sc=None)` makes the session optional and resolves the active one inside; this Connect signature is `getOrCreate(sparkSession)` — positional, required. Code that does `SQLContext.getOrCreate()` after importing from `pyspark.sql.connect.context` directly hits a `TypeError`.
(b) The body of this method is identical to `_get_or_create_from_session` except for `warnings.warn` — it could just call through.
Both concerns stop mattering if the class is made implementation-private (Finding #1).
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
Review Comment:
Sentence is incomplete — inferring what?
```suggestion
the sample ratio of rows used for inferring the schema.
```
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ rows = [
Review Comment:
Worth a one-line comment that this materializes the catalog list on the driver (chosen over `SHOW TABLES` for column-name parity per the earlier @Yicong-Huang thread). For catalogs with very many tables this is in-memory work that classic delegates server-side; future readers will appreciate the rationale being captured here rather than only in PR history.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ rows = [
+ (t.namespace[0] if t.namespace else "", t.name, t.isTemporary)
+ for t in self.sparkSession.catalog.listTables(dbName)
+ ]
+ return self.sparkSession.createDataFrame(rows, schema)
+
+ def tableNames(self, dbName: Optional[str] = None) -> List[str]:
+ """Returns a list of names of tables in the database ``dbName``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str
+ name of the database to use. Default to the current database.
+
+ Returns
+ -------
+ list
+ list of table names, in string
+ """
+ return [t.name for t in self.sparkSession.catalog.listTables(dbName)]
+
+ def cacheTable(self, tableName: str) -> None:
+ """Caches the specified table in-memory.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.cacheTable(tableName)
+
+ def uncacheTable(self, tableName: str) -> None:
+ """Removes the specified table from the in-memory cache.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.uncacheTable(tableName)
+
+ def clearCache(self) -> None:
+ """Removes all cached tables from the in-memory cache.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.clearCache()
+
+ @property
+ def read(self) -> DataFrameReader:
+ """Returns a :class:`DataFrameReader` that can be used to read data
+ in as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrameReader`
+ """
+ return self.sparkSession.read
+
+ @property
+ def readStream(self) -> DataStreamReader:
+ """Returns a :class:`DataStreamReader` that can be used to read data streams
+ as a streaming :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Notes
+ -----
+ This API is evolving.
+
+ Returns
+ -------
+ :class:`DataStreamReader`
+ """
+ return self.sparkSession.readStream
+
+ @property
+ def streams(self) -> StreamingQueryManager:
+ """Returns a :class:`StreamingQueryManager` that allows managing all the
+ :class:`~pyspark.sql.streaming.StreamingQuery` StreamingQueries active on
Review Comment:
"StreamingQuery StreamingQueries" duplicates the class name; the backticks around `this` read oddly:
```suggestion
:class:`~pyspark.sql.streaming.StreamingQuery` instances active on this
```
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
Review Comment:
The signature accepts `Tuple[str, ...]` too:
```suggestion
a :class:`~pyspark.sql.types.DataType` or a datatype string or a list/tuple of
```
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ rows = [
+ (t.namespace[0] if t.namespace else "", t.name, t.isTemporary)
+ for t in self.sparkSession.catalog.listTables(dbName)
+ ]
+ return self.sparkSession.createDataFrame(rows, schema)
+
+ def tableNames(self, dbName: Optional[str] = None) -> List[str]:
+ """Returns a list of names of tables in the database ``dbName``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str
+ name of the database to use. Default to the current database.
+
+ Returns
+ -------
+ list
+ list of table names, in string
Review Comment:
Awkward phrasing:
```suggestion
list of table names as strings
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3292250725
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning.
+
+ Used internally by :meth:`newSession` and :meth:`getOrCreate`.
+ """
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 1.6.0
+ """
+ return self._from_session(self.sparkSession.newSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 1.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 1.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 1.3.1
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 3.5.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 1.4.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 1.2.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 2.1.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame( # type: ignore[call-overload]
+ data, schema, samplingRatio, verifySchema
+ )
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 1.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 1.6.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 1.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 1.3.0
Review Comment:
might need to update the version again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4713501246
I think there are two minor points:
1. `SparkConnectClient.newSession()` reaches into the private `_params` dict. `newSession()` drops the session id with `new_connection._params.pop(ChannelBuilder.PARAM_SESSION_ID, None)`, whereas the sibling clone() uses the public `new_connection.set(...)`. Maybe we can use public api for `newSession` as well.
2. In `SQLContext.getOrCreate()`, the `is_remote()` branch obtains the session through the classic `SparkSession._getActiveSessionOrCreate()` and then casts it to a Connect SparkSession before wrapping it. I am not sure if this is intended behavior, @dbtsai please confirm.
I think both can be fixed in follow ups. If no objections I will merge this PR tonight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306184477
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,37 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect.context import SQLContext as ConnectSQLContext
+
+ session = SparkSession._getActiveSessionOrCreate()
+ return ConnectSQLContext._get_or_create_from_session( # type: ignore[return-value]
+ session # type: ignore[arg-type]
Review Comment:
Replace with `cast()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3331756052
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,459 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.2.0
Review Comment:
Fixed in d6c985b9334 — changed all 22 occurrences to `4.3.0`.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/tests/test_sql_context.py:
##########
@@ -0,0 +1,137 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import warnings
+
+from pyspark import SQLContext
+from pyspark.sql import SparkSession
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class SQLContextTestsMixin:
+ """Tests for SQLContext that run in both classic and Connect modes.
+
+ Subclasses must implement :meth:`_make_ctx` to return an :class:`SQLContext`
+ appropriate for their mode, and must expose ``self.spark`` as a
+ :class:`SparkSession`.
+ """
+
+ spark: SparkSession
+
+ def _make_ctx(self) -> SQLContext:
+ raise NotImplementedError
+
+ def setUp(self) -> None:
+ SQLContext._instantiatedContext = None
+
+ def tearDown(self) -> None:
+ SQLContext._instantiatedContext = None
+
+ def test_setConf_and_getConf(self) -> None:
+ ctx = self._make_ctx()
+ ctx.setConf("spark.sql.shuffle.partitions", "42")
+ self.assertEqual(ctx.getConf("spark.sql.shuffle.partitions"), "42")
+ self.assertEqual(ctx.getConf("spark.sql.shuffle.partitions", "99"), "42")
+ ctx.setConf("spark.sql.shuffle.partitions", "200")
+
+ def test_range(self) -> None:
+ ctx = self._make_ctx()
+ self.assertEqual(ctx.range(3).count(), 3)
+
+ def test_createDataFrame(self) -> None:
+ ctx = self._make_ctx()
+ df = ctx.createDataFrame([(1, "a"), (2, "b")], ["id", "val"])
+ self.assertEqual(df.count(), 2)
+
+ def test_sql(self) -> None:
+ ctx = self._make_ctx()
+ self.assertEqual(ctx.sql("SELECT 1 AS n").collect()[0].n, 1)
+
+ def test_registerDataFrameAsTable_and_sql(self) -> None:
+ ctx = self._make_ctx()
+ df = self.spark.createDataFrame([(1, "r1"), (2, "r2")], ["field1", "field2"])
+ ctx.registerDataFrameAsTable(df, "ctx_mixin_tbl")
+ try:
+ result = ctx.sql("SELECT field1 FROM ctx_mixin_tbl ORDER BY field1").collect()
+ self.assertEqual([r.field1 for r in result], [1, 2])
+ finally:
+ ctx.dropTempTable("ctx_mixin_tbl")
+
+ def test_table(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(42,)], ["v"]).createOrReplaceTempView("ctx_mixin_view")
+ try:
+ self.assertEqual(ctx.table("ctx_mixin_view").collect()[0].v, 42)
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_view")
+
+ def test_tables_contains_registered_view(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView("ctx_mixin_tables_view")
+ try:
+ names = [r.tableName for r in ctx.tables().collect()]
+ self.assertIn("ctx_mixin_tables_view", names)
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_tables_view")
+
+ def test_tableNames_contains_registered_view(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView(
+ "ctx_mixin_tablenames_view"
+ )
+ try:
+ self.assertIn("ctx_mixin_tablenames_view", ctx.tableNames())
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_tablenames_view")
+
+ def test_cacheTable_and_uncacheTable(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView("ctx_mixin_cache_view")
+ try:
+ ctx.cacheTable("ctx_mixin_cache_view")
+ ctx.uncacheTable("ctx_mixin_cache_view")
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_cache_view")
+
+ def test_clearCache(self) -> None:
+ self._make_ctx().clearCache()
+
+ def test_newSession_returns_distinct_instance(self) -> None:
+ ctx = self._make_ctx()
+ ctx2 = ctx.newSession()
+ self.assertIsNot(ctx, ctx2)
+
+ def test_read_is_available(self) -> None:
+ self.assertIsNotNone(self._make_ctx().read)
+
+ def test_readStream_is_available(self) -> None:
+ self.assertIsNotNone(self._make_ctx().readStream)
+
+ def test_streams_is_available(self) -> None:
Review Comment:
Fixed in d6c985b9334 — added `test_udf_is_available`, `test_udtf_is_available`, and `test_registerFunction` to `SQLContextTestsMixin`. `createExternalTable` is excluded from the mixin (it requires a real external path and is covered by the classic test suite separately); updated the PR description to match.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,459 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
Review Comment:
Intentional — added a note to `_get_or_create_from_session` in d6c985b9334 explaining the design: Connect sessions have no JVM lifecycle sentinel (no `_sc._jsc`) to inspect, and the cache is already cleared by the `stop()` hook when the session terminates. Re-validating on every call would require a round-trip to the server.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3370150462
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.3.0
+
+ .. deprecated:: 2.3.0
Review Comment:
I think deprecation should also be 4.3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3150660881
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning.
+
+ Used internally by :meth:`newSession` and :meth:`getOrCreate`.
+ """
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 1.6.0
+ """
+ return self._from_session(self.sparkSession.newSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 1.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 1.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 1.3.1
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 3.5.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 1.4.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 1.2.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 2.1.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame( # type: ignore[call-overload]
+ data, schema, samplingRatio, verifySchema
+ )
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 1.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 1.6.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 1.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 1.3.0
Review Comment:
Should fix the version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306181226
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ if dbName is None:
+ return self.sparkSession.sql("SHOW TABLES")
+ return self.sparkSession.sql(f"SHOW TABLES IN {dbName}")
Review Comment:
Yes, fix with `catalog.listTables()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306217879
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
Review Comment:
Good catch — confirmed this was a real bypass. `HiveContext.getOrCreate(spark)` dispatches to `SQLContext.getOrCreate(cls=HiveContext, ...)`, which calls `_from_session` → `object.__new__(HiveContext)`, creating an instance without ever invoking `__init__`, so the `PySparkNotImplementedError` was silently skipped.
Fixed in c967950ec9e by overriding `_from_session` in `HiveContext` to unconditionally raise. Since `getOrCreate`, `_get_or_create_from_session`, and `newSession` all route through `_from_session`, this closes all bypass paths. Added `test_hive_context_get_or_create_raises` to cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3383938300
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
Review Comment:
You're right -- thanks. Returning a clone was wrong; `newSession()` should hand back a fresh, independent session with empty state.
Fixed in aaa51c8a4bc (with a follow-up cleanup in 3c293404fc6). The root cause was that the Connect `SparkSession` only had `cloneSession()` (which copies conf / temp views / registered functions / catalog state) and no equivalent of classic `newSession()`. I added `SparkSession.newSession()` to the Connect `SparkSession`, mirroring the existing Scala Connect `newSession()` in `SparkSession.scala`: it rebuilds the client against the same endpoint with the session id cleared, so a fresh UUID is generated and the server lazily creates an empty, isolated session -- no `CloneSession` RPC and no state copy. `SQLContext.newSession()` now delegates to that instead of `cloneSession()`, the docstring is back to the classic contract (separate `SQLConf` / temp views / UDFs, shared table cache), and the test asserts the parent's temp views are not visible in the new session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3150659978
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
Review Comment:
We do sth in SQLContext.getOrCreate. Otherwise, users would have to manually import pyspark.sql.connect to use spark Connect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3321010090
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,39 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect.context import SQLContext as ConnectSQLContext
Review Comment:
Fixed via your option (a): the classic `SQLContext.getOrCreate` now does `connect_cls = getattr(_connect_context, cls.__name__, _connect_context.SQLContext)` before calling `_get_or_create_from_session`. When `cls` is `HiveContext`, this routes to `ConnectHiveContext._get_or_create_from_session`, which calls `ConnectHiveContext._from_session` — and that raises `PySparkNotImplementedError`.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
Review Comment:
Fixed: updated the `newSession()` docstring to accurately describe `cloneSession()` semantics — it creates a new independent server session with the current session's configuration, temporary views, and registered functions copied in, rather than returning a fresh-empty session.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
Review Comment:
Fixed as a consequence of Finding #1: with `getOrCreate` removed from the Connect class, both (a) the signature divergence and (b) the duplication disappear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3327426530
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,459 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
Review Comment:
Cached instance is never validated against the currently-active session - it only resets via the `stop()` hook. If a different Connect session becomes active after the first `getOrCreate()`, this keeps returning a context bound to the old session. Classic `_get_or_create` re-creates when the cached context is dead. Intentional?
##########
python/pyspark/sql/tests/test_sql_context.py:
##########
@@ -0,0 +1,137 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import warnings
+
+from pyspark import SQLContext
+from pyspark.sql import SparkSession
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class SQLContextTestsMixin:
+ """Tests for SQLContext that run in both classic and Connect modes.
+
+ Subclasses must implement :meth:`_make_ctx` to return an :class:`SQLContext`
+ appropriate for their mode, and must expose ``self.spark`` as a
+ :class:`SparkSession`.
+ """
+
+ spark: SparkSession
+
+ def _make_ctx(self) -> SQLContext:
+ raise NotImplementedError
+
+ def setUp(self) -> None:
+ SQLContext._instantiatedContext = None
+
+ def tearDown(self) -> None:
+ SQLContext._instantiatedContext = None
+
+ def test_setConf_and_getConf(self) -> None:
+ ctx = self._make_ctx()
+ ctx.setConf("spark.sql.shuffle.partitions", "42")
+ self.assertEqual(ctx.getConf("spark.sql.shuffle.partitions"), "42")
+ self.assertEqual(ctx.getConf("spark.sql.shuffle.partitions", "99"), "42")
+ ctx.setConf("spark.sql.shuffle.partitions", "200")
+
+ def test_range(self) -> None:
+ ctx = self._make_ctx()
+ self.assertEqual(ctx.range(3).count(), 3)
+
+ def test_createDataFrame(self) -> None:
+ ctx = self._make_ctx()
+ df = ctx.createDataFrame([(1, "a"), (2, "b")], ["id", "val"])
+ self.assertEqual(df.count(), 2)
+
+ def test_sql(self) -> None:
+ ctx = self._make_ctx()
+ self.assertEqual(ctx.sql("SELECT 1 AS n").collect()[0].n, 1)
+
+ def test_registerDataFrameAsTable_and_sql(self) -> None:
+ ctx = self._make_ctx()
+ df = self.spark.createDataFrame([(1, "r1"), (2, "r2")], ["field1", "field2"])
+ ctx.registerDataFrameAsTable(df, "ctx_mixin_tbl")
+ try:
+ result = ctx.sql("SELECT field1 FROM ctx_mixin_tbl ORDER BY field1").collect()
+ self.assertEqual([r.field1 for r in result], [1, 2])
+ finally:
+ ctx.dropTempTable("ctx_mixin_tbl")
+
+ def test_table(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(42,)], ["v"]).createOrReplaceTempView("ctx_mixin_view")
+ try:
+ self.assertEqual(ctx.table("ctx_mixin_view").collect()[0].v, 42)
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_view")
+
+ def test_tables_contains_registered_view(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView("ctx_mixin_tables_view")
+ try:
+ names = [r.tableName for r in ctx.tables().collect()]
+ self.assertIn("ctx_mixin_tables_view", names)
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_tables_view")
+
+ def test_tableNames_contains_registered_view(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView(
+ "ctx_mixin_tablenames_view"
+ )
+ try:
+ self.assertIn("ctx_mixin_tablenames_view", ctx.tableNames())
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_tablenames_view")
+
+ def test_cacheTable_and_uncacheTable(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView("ctx_mixin_cache_view")
+ try:
+ ctx.cacheTable("ctx_mixin_cache_view")
+ ctx.uncacheTable("ctx_mixin_cache_view")
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_cache_view")
+
+ def test_clearCache(self) -> None:
+ self._make_ctx().clearCache()
+
+ def test_newSession_returns_distinct_instance(self) -> None:
+ ctx = self._make_ctx()
+ ctx2 = ctx.newSession()
+ self.assertIsNot(ctx, ctx2)
+
+ def test_read_is_available(self) -> None:
+ self.assertIsNotNone(self._make_ctx().read)
+
+ def test_readStream_is_available(self) -> None:
+ self.assertIsNotNone(self._make_ctx().readStream)
+
+ def test_streams_is_available(self) -> None:
Review Comment:
Mixin has no coverage for `udf`, `udtf`, `registerFunction`, or `createExternalTable`, yet the PR description claims `udf`/`udtf`/`registerFunction` are covered. please consider add the cases or correct the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3151877052
##########
python/pyspark/sql/tests/connect/test_connect_context.py:
##########
@@ -0,0 +1,187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+import warnings
+
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.context import HiveContext, SQLContext
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class SQLContextConnectTests(ReusedConnectTestCase):
Review Comment:
SQLContextTestsMixin extracted into test_context.py; Connect test is now SQLContextParityTests(SQLContextTestsMixin, ReusedConnectTestCase)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4673013304
cc @hvanhovell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3383935252
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 4.3.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 4.3.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
Review Comment:
Thanks Wenchen. I went the other way on this one -- returning a genuinely fresh session rather than raising -- and I think the parity argument settles it: Scala Connect `SparkSession` already supports `newSession()` (`sql/connect/common/.../SparkSession.scala:403`), implemented as `SparkSession.builder().client(client.copy()).create()` -- a fresh, independent session bound to the same connection with no state copied in. So this isn't a deliberate cross-language "Connect can't do `newSession`" stance; the Python side was just missing the construct Scala already has.
Pushed in 3c293404fc6 / aaa51c8a4bc:
- Added `SparkSession.newSession()` to the Connect Python `SparkSession`, mirroring Scala: it rebuilds the client against the same endpoint with the session id cleared, so a fresh UUID is generated and the server lazily creates an empty isolated session -- no `CloneSession` RPC, no state copy, and no dependency on the `cloneSession` developer API.
- Removed the now-dead `"newSession"` entry from the `__getattr__` `JVM_ATTRIBUTE_NOT_SUPPORTED` guard (the real method shadows it anyway).
- `SQLContext.newSession()` delegates to it; the test now asserts the parent's temp views are not visible in the new session.
This gives @hvanhovell the independent-session semantics he asked for, matches classic `newSession()`, and brings Python Connect to parity with Scala Connect rather than diverging by raising. Happy to revisit if you'd still prefer raising for the deprecated shim specifically, but raising would make `df.sparkSession.newSession()` behave differently across the Scala and Python Connect clients. I'll also drop the "the Connect equivalent of `SparkSession.newSession()`" wording from the PR description regardless.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 4.3.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 4.3.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.3.0
+
+ .. deprecated:: 4.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 4.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.3.0
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list/tuple of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring the schema.
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ # Use catalog.listTables() rather than SHOW TABLES so the column names are always
+ # (namespace, tableName, isTemporary), matching the classic implementation.
+ # SHOW TABLES returns "database" vs "namespace" depending on the active catalog.
+ rows = [
+ (t.namespace[-1] if t.namespace else "", t.name, t.isTemporary)
Review Comment:
Good catch -- fixed in 3c293404fc6. Switched `t.namespace[-1]` to `".".join(t.namespace)` so multi-level v2 catalog namespaces read `a.b` like classic SHOW TABLES instead of dropping to `b`. Added a comment noting the parity rationale.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
Review Comment:
Fixed in 3c293404fc6 -- replaced the em-dash with an ASCII hyphen. Confirmed with `grep -P "[^\x00-\x7F]"` that the file is now ASCII-only.
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +143,51 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.sql.utils import is_remote
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote():
+ from pyspark.sql.connect import context as _connect_context
+ from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
+
+ session = SparkSession._getActiveSessionOrCreate()
+ # Route to the Connect counterpart so subclasses (e.g. HiveContext) are handled
+ # correctly: the Connect HiveContext._from_session raises PySparkNotImplementedError.
+ connect_cls = getattr(_connect_context, cls.__name__, _connect_context.SQLContext)
Review Comment:
Fixed in 3c293404fc6. The dispatch now does `getattr(_connect_context, cls.__name__, None)` and raises `PySparkNotImplementedError` when a subclass has no Connect counterpart, instead of silently handing back a base Connect `SQLContext`. `HiveContext` is unaffected (it has a Connect counterpart and is still routed by name to raise via `_from_session`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4694298900
cc @viirya @cloud-fan @Yicong-Huang @hvanhovell @HyukjinKwon to review again since many changes after the approvals. Thanks,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3375583533
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.3.0
+
+ .. deprecated:: 2.3.0
Review Comment:
@dbtsai shall we address it before merging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4624977617
Thanks for tracing through it so carefully, @viirya — and for confirming the Connect-dispatch and `HiveContext` reasoning. Addressed in 1b105b6ca61:
- **`assert sc is not None` (blocking).** Replaced with an explicit `PySparkValueError(errorClass="ARGUMENT_REQUIRED", ...)` so the guard holds under `python -O`. Good call — the stripped-assert path would have failed with a cryptic `AttributeError` on `sc._jvm` deep in `_get_or_create`.
- **Duplicate `SQLContextTests` class name.** Renamed the new mixin-based classic suite to `SQLContextClassicTests`. Kept the existing `SQLContextTests` smoke test in `test_context.py` since it covers a distinct path (`getOrCreate(sc)` against a real local `SparkContext`, which the constructor-based mixin doesn't).
- **`legacy.rst` note.** Added a `.. note::` calling out that `registerJavaFunction` and `HiveContext` are unsupported under Spark Connect and raise `PySparkNotImplementedError`.
- **`newSession` inherited-state test.** Added `test_newSession_inherits_state` to the Connect-specific suite (`test_connect_context.py`) — it registers a temp view, calls `newSession()`, and asserts the clone sees it, locking in the documented `cloneSession` inherited-state behavior. It lives in the Connect-only suite rather than the shared mixin precisely because this semantics intentionally differs from classic `newSession` (fresh session, shared-cache-only), so it can't be a parity assertion.
CI was green on the previous revision; I'll keep an eye on this run as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4664547513
Ran an additional automated + manual review pass over this PR. Summary of findings and follow-up fixes (commit incoming):
### 1. Critical: sessions created via `object.__new__` are missing `release_session_on_close`
`newSession()` (added in this PR) builds the new `SparkSession` with `object.__new__(SparkSession)` and only sets `_client` and `_session_id`. Since `__init__` is bypassed, `release_session_on_close` (set at `session.py:312`) is never set, and `stop()` reads it unconditionally at `session.py:951` - so calling `stop()` on a session returned by `newSession()` raises `AttributeError` and breaks the normal cleanup path.
`cloneSession()` has the identical pre-existing bug (it predates this PR), so the fix covers both:
- Set `release_session_on_close = True` in both `newSession()` and `cloneSession()`.
- Regression assertions added in `test_connect_context.py` and `test_connect_clone_session.py` (the local-remote test harness cannot safely call `stop()` on a child session, so the tests assert the attribute directly).
### 2. Connect client leak in the parity test
`SQLContextTestsMixin.test_newSession_returns_distinct_instance` runs under `SQLContextParityTests(ReusedConnectTestCase)`, where `ctx.newSession()` creates a brand-new `SparkConnectClient` (which registers an atexit hook at `core.py:760`) that the test never releases or closes. This is exactly the hang-after-`tearDownClass` failure mode already documented in `test_connect_context.py`. Fixed by overriding the test in the Connect parity subclass with the same `release_session()` + `close()` cleanup; the classic mixin is untouched.
### 3. Reviewed and rejected: "newSession() drops client settings that clone() preserves"
A reviewer tool suggested `SparkConnectClient.newSession()` should pass through `session_hooks` / retry settings like the clone path. This is not accurate: `clone()` constructs its new client with exactly the same three arguments (`connection` = deepcopy of the builder, `user_id`, `use_reattachable_execute`). The builder deepcopy already preserves the endpoint, channel options, and metadata, and `session_hooks` cannot be propagated since hook factories are consumed in `SparkSession.__init__` and bound to that specific session. `newSession()` intentionally mirrors the `clone()` constructor pattern; any improvement there is a shared, pre-existing concern and out of scope here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3326461092
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,457 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
Review Comment:
Finding #1 made these classes an internal implementation detail, but they're still exported in `__all__`, which signals public API. If they're meant to be internal, drop them from `__all__` — the classic dispatch in `context.py` imports the module attribute directly (`getattr(_connect_context, cls.__name__, ...)`), so it doesn't rely on `__all__`. Non-blocking.
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,44 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
Review Comment:
This `is_remote_only()` dispatch is the PR's central user-facing behavior, and the Finding #2 fix routes `HiveContext.getOrCreate()` here to raise — but neither is tested. The reorg dropped `test_getOrCreate_emits_deprecation_warning` and `test_hive_context_get_or_create_raises` (both present at the prior review commit `6af283f`, the latter added specifically to cover Finding #2) without replacing them. `is_remote_only()` is mockable (`pyspark.util.is_remote_only`), so a `ReusedConnectTestCase` can patch it `True` and assert (1) `SQLContext.getOrCreate()` returns a Connect-backed context and (2) `HiveContext.getOrCreate()` raises `PySparkNotImplementedError`.
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -27,6 +27,7 @@
from pyspark.sql import Row, SparkSession
from pyspark.sql.types import StructType, StringType, StructField
from pyspark.testing.sqlutils import ReusedSQLTestCase
+from pyspark.sql.tests.test_sql_context import SQLContextTestsMixin # noqa: F401
Review Comment:
This import is dead — `SQLContextTestsMixin` isn't a `TestCase` and isn't subclassed in this file, so it's never collected or run (hence the `# noqa: F401`). Looks like a leftover from the reorg; please remove it.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,457 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.0.0
Review Comment:
Confirming @Yicong-Huang's earlier question: the repo is at `5.0.0.dev0` and 4.0.0 is already released, so these brand-new methods can't be `versionadded:: 4.0.0` — they should be `5.0.0`. Applies to every `.. versionadded:: 4.0.0` in this file (lines 107, 120, 133, 141, 153, 172, 197, 210, …).
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,44 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect import context as _connect_context
+ from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
+
+ session = SparkSession._getActiveSessionOrCreate()
+ # Route to the Connect counterpart so subclasses (e.g. HiveContext) are handled
+ # correctly: ConnectHiveContext._from_session raises PySparkNotImplementedError.
Review Comment:
There's no class named `ConnectHiveContext` — the Connect-side class is `HiveContext` (in `pyspark.sql.connect.context`), and the routing reaches it via `cls.__name__`.
```suggestion
# correctly: the Connect HiveContext._from_session raises PySparkNotImplementedError.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4619382519
Thanks for the careful review, @viirya! Addressed in b51a75480d0:
1. **`SQLContext(spark)` constructor vs `getOrCreate()`.** Narrowed the scope rather than wiring the constructor — this keeps the Connect `SQLContext` an internal implementation detail (per @cloud-fan's Finding #1) instead of re-exposing a public Connect constructor. Updated the PR description and the change notes to state explicitly that only `SQLContext.getOrCreate()` is wired to Connect; `SQLContext(...)` stays classic-only and still requires a `SparkContext`. The public path through `from pyspark.sql import SQLContext` is already covered by `test_getOrCreate_emits_deprecation_and_returns_connect_context` (imports `from pyspark.sql import SQLContext` and calls `getOrCreate()`).
2. **`is_remote_only()` → `is_remote()`.** Done. `getOrCreate()` now gates on `is_remote()` (`pyspark.sql.utils`), so a full PySpark install talking to a Connect server via `SPARK_REMOTE` is covered, not just the remote-only pyspark-client packaging. Thanks for tracing the resolve path — confirmed `_getActiveSessionOrCreate()` lands on the active Connect session, so the cast stays valid. Test patch targets updated to `pyspark.sql.utils.is_remote`.
3. **Mixin `setUp`/`tearDown` super() chain.** Fixed — `SQLContextTestsMixin.setUp` now calls `super().setUp()` first and `tearDown` calls `super().tearDown()` last, so `ReusedSQLTestCase` / `ReusedConnectTestCase` ML-cache and worker-log cleanup runs in both `SQLContextTests` and `SQLContextParityTests`.
4. **`legacy.rst` `.. deprecated::` version.** Fixed — now `.. deprecated:: 3.0.0` to match the class docstring.
5. **`session.py` `stop()` unconditional import.** Fixed — guarded behind `sys.modules.get("pyspark.sql.connect.context")`, so if no `SQLContext` was ever created (module never imported) there's nothing to reset and we don't force the import.
And thanks for confirming the wired parts (`tables()` via `catalog.listTables()`, the `HiveContext` `getOrCreate` routing, and the Connect `HiveContext._instantiatedContext` override).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4715840937
Thanks all, merged to master/4.x. Let's handle minor issues in followups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3331952282
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,459 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
Review Comment:
Fixed properly in 74be8e457de — `_get_or_create_from_session` now re-creates the context whenever `sparkSession is not cls._instantiatedContext.sparkSession`, directly mirroring the classic `_sc._jsc is None` guard. If the previous session is stopped and a new one started, the next `getOrCreate()` call will return a fresh context bound to the new session.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
##########
python/pyspark/sql/tests/test_sql_context.py:
##########
@@ -0,0 +1,137 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import warnings
+
+from pyspark import SQLContext
+from pyspark.sql import SparkSession
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class SQLContextTestsMixin:
+ """Tests for SQLContext that run in both classic and Connect modes.
+
+ Subclasses must implement :meth:`_make_ctx` to return an :class:`SQLContext`
+ appropriate for their mode, and must expose ``self.spark`` as a
+ :class:`SparkSession`.
+ """
+
+ spark: SparkSession
+
+ def _make_ctx(self) -> SQLContext:
+ raise NotImplementedError
+
+ def setUp(self) -> None:
+ SQLContext._instantiatedContext = None
+
+ def tearDown(self) -> None:
+ SQLContext._instantiatedContext = None
+
+ def test_setConf_and_getConf(self) -> None:
+ ctx = self._make_ctx()
+ ctx.setConf("spark.sql.shuffle.partitions", "42")
+ self.assertEqual(ctx.getConf("spark.sql.shuffle.partitions"), "42")
+ self.assertEqual(ctx.getConf("spark.sql.shuffle.partitions", "99"), "42")
+ ctx.setConf("spark.sql.shuffle.partitions", "200")
+
+ def test_range(self) -> None:
+ ctx = self._make_ctx()
+ self.assertEqual(ctx.range(3).count(), 3)
+
+ def test_createDataFrame(self) -> None:
+ ctx = self._make_ctx()
+ df = ctx.createDataFrame([(1, "a"), (2, "b")], ["id", "val"])
+ self.assertEqual(df.count(), 2)
+
+ def test_sql(self) -> None:
+ ctx = self._make_ctx()
+ self.assertEqual(ctx.sql("SELECT 1 AS n").collect()[0].n, 1)
+
+ def test_registerDataFrameAsTable_and_sql(self) -> None:
+ ctx = self._make_ctx()
+ df = self.spark.createDataFrame([(1, "r1"), (2, "r2")], ["field1", "field2"])
+ ctx.registerDataFrameAsTable(df, "ctx_mixin_tbl")
+ try:
+ result = ctx.sql("SELECT field1 FROM ctx_mixin_tbl ORDER BY field1").collect()
+ self.assertEqual([r.field1 for r in result], [1, 2])
+ finally:
+ ctx.dropTempTable("ctx_mixin_tbl")
+
+ def test_table(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(42,)], ["v"]).createOrReplaceTempView("ctx_mixin_view")
+ try:
+ self.assertEqual(ctx.table("ctx_mixin_view").collect()[0].v, 42)
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_view")
+
+ def test_tables_contains_registered_view(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView("ctx_mixin_tables_view")
+ try:
+ names = [r.tableName for r in ctx.tables().collect()]
+ self.assertIn("ctx_mixin_tables_view", names)
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_tables_view")
+
+ def test_tableNames_contains_registered_view(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView(
+ "ctx_mixin_tablenames_view"
+ )
+ try:
+ self.assertIn("ctx_mixin_tablenames_view", ctx.tableNames())
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_tablenames_view")
+
+ def test_cacheTable_and_uncacheTable(self) -> None:
+ ctx = self._make_ctx()
+ self.spark.createDataFrame([(1,)], ["x"]).createOrReplaceTempView("ctx_mixin_cache_view")
+ try:
+ ctx.cacheTable("ctx_mixin_cache_view")
+ ctx.uncacheTable("ctx_mixin_cache_view")
+ finally:
+ self.spark.catalog.dropTempView("ctx_mixin_cache_view")
+
+ def test_clearCache(self) -> None:
+ self._make_ctx().clearCache()
+
+ def test_newSession_returns_distinct_instance(self) -> None:
+ ctx = self._make_ctx()
+ ctx2 = ctx.newSession()
+ self.assertIsNot(ctx, ctx2)
+
+ def test_read_is_available(self) -> None:
+ self.assertIsNotNone(self._make_ctx().read)
+
+ def test_readStream_is_available(self) -> None:
+ self.assertIsNotNone(self._make_ctx().readStream)
+
+ def test_streams_is_available(self) -> None:
Review Comment:
Fixed in 74be8e457de — added `test_createExternalTable` to `SQLContextTestsMixin` (writes a small Parquet dataset to a temp directory, calls `ctx.createExternalTable`, asserts the row count). PR description updated to include `createExternalTable` in the covered list.
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3376053998
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.3.0
+
+ .. deprecated:: 2.3.0
Review Comment:
Good catch — fixed in 982c0fa032f. Bumped all three `.. deprecated::` directives (the class, `registerFunction`, and `HiveContext`) plus the two matching runtime `FutureWarning` strings to `4.3.0`, consistent with the `.. versionadded:: 4.3.0` directives. A method cannot be deprecated before the version it was added in, so these now match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3150656795
##########
python/pyspark/sql/tests/connect/test_connect_context.py:
##########
@@ -0,0 +1,187 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+import warnings
+
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.context import HiveContext, SQLContext
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class SQLContextConnectTests(ReusedConnectTestCase):
Review Comment:
We should share the tests with Spark Classic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3151877735
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning.
+
+ Used internally by :meth:`newSession` and :meth:`getOrCreate`.
+ """
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 1.6.0
+ """
+ return self._from_session(self.sparkSession.newSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 1.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 1.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 1.3.1
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 3.5.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 1.4.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 1.2.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 2.1.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame( # type: ignore[call-overload]
+ data, schema, samplingRatio, verifySchema
+ )
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 1.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 1.6.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 1.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 1.3.0
Review Comment:
All versionadded changed to 4.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3321009403
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -171,6 +171,109 @@ def range_frame_match():
reload(window)
+class SQLContextTestsMixin:
Review Comment:
Fixed: created `python/pyspark/sql/tests/test_sql_context.py` with `SQLContextTestsMixin` + `SQLContextTests(SQLContextTestsMixin, ReusedSQLTestCase)` so the mixin now runs against classic Spark too. Added `python/pyspark/sql/tests/connect/test_parity_sql_context.py` following the existing Connect parity convention (`SQLContextParityTests(SQLContextTestsMixin, ReusedConnectTestCase)`). `test_connect_context.py` is now Connect-specific tests only.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
Review Comment:
Fixed via option (a): removed the public `getOrCreate` classmethod from the Connect class entirely. The Connect `SQLContext` is now an internal implementation detail, accessible only through `_get_or_create_from_session` (invoked by the classic dispatch). Legacy code calling `SQLContext.getOrCreate()` via the classic import path continues to work unchanged; new code uses `SparkSession`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3327377685
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,457 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.0.0
Review Comment:
Changed to `versionadded:: 4.2.0` as claude code doesn't have context about our new release model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3308937672
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -171,6 +171,109 @@ def range_frame_match():
reload(window)
+class SQLContextTestsMixin:
Review Comment:
this is just a mixin, not a real test, so it is not tested in classic sides.
I think we can reorg the tests in this way to following existing connect parity test convention
1, python/pyspark/sql/tests/test_sql_context.py
- class SQLContextTestsMixin
- class SQLContextTests(SQLContextTestsMixin,ReusedSQLTestCase)
2, python/pyspark/sql/tests/connect/test_parity_sql_context.py
- class SQLContextParityTests(SQLContextTestsMixin,ReusedConnectTestCase)
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -171,6 +171,109 @@ def range_frame_match():
reload(window)
+class SQLContextTestsMixin:
Review Comment:
this is just a mixin, not a real test, so it is not tested in classic sides.
I think we can reorg the tests in this way to follow existing connect parity test convention
1, python/pyspark/sql/tests/test_sql_context.py
- class SQLContextTestsMixin
- class SQLContextTests(SQLContextTestsMixin,ReusedSQLTestCase)
2, python/pyspark/sql/tests/connect/test_parity_sql_context.py
- class SQLContextParityTests(SQLContextTestsMixin,ReusedConnectTestCase)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3292273496
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ if dbName is None:
+ return self.sparkSession.sql("SHOW TABLES")
+ return self.sparkSession.sql(f"SHOW TABLES IN {dbName}")
Review Comment:
might worth to check on parity, the classic returns columns `(namespace, tableName, isTemporary)`, while `SHOW TABLES` returns `(namespace, tableName, isTemporary)` on some catalogs and `(database, tableName,
isTemporary)` on others.
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,37 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect.context import SQLContext as ConnectSQLContext
+
+ session = SparkSession._getActiveSessionOrCreate()
+ return ConnectSQLContext._get_or_create_from_session( # type: ignore[return-value]
+ session # type: ignore[arg-type]
Review Comment:
nit: is it possible to get rid of this type ignore?
##########
python/pyspark/sql/tests/test_context.py:
##########
@@ -171,6 +171,109 @@ def range_frame_match():
reload(window)
+class SQLContextTestsMixin:
+ """Tests for SQLContext that run in both classic and Connect modes.
+
+ Subclasses must implement :meth:`_make_ctx` to return an :class:`SQLContext`
+ appropriate for their mode, and must expose ``self.spark`` as a
+ :class:`SparkSession`.
+ """
+
+ spark: SparkSession
+
+ def _make_ctx(self) -> SQLContext:
+ raise NotImplementedError
+
+ def setUp(self) -> None:
+ SQLContext._instantiatedContext = None
Review Comment:
I think this is
`pyspark.sql.context.SQLContext`, not `pyspark.sql.connect.context.SQLContext`?
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
Review Comment:
can `HiveContext.getOrCreate(spark)` work? it might be able to bypass the not supported `HiveContext`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3151875949
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning.
+
+ Used internally by :meth:`newSession` and :meth:`getOrCreate`.
+ """
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 1.6.0
+ """
+ return self._from_session(self.sparkSession.newSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 1.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 1.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 1.3.1
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 3.5.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 1.4.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 1.2.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 2.1.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame( # type: ignore[call-overload]
+ data, schema, samplingRatio, verifySchema
+ )
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 1.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 1.6.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 1.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ listed = self.sparkSession.catalog.listTables(dbName)
+ rows = [
Review Comment:
Replaced with spark.sql("SHOW TABLES [IN dbName]")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3151874599
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
Review Comment:
Classic getOrCreate now detects is_remote_only() and dispatches to ConnectSQLContext._get_or_create_from_session(); sc made optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3382698337
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,467 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
Review Comment:
I think that is incorrect. This needs to return a new independent session, not a clone...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3383421893
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 4.3.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 4.3.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
Review Comment:
Reinforcing @hvanhovell's comment with what I traced — I accepted the docstring-only fix last round, and I now think that was a miss. Classic `newSession()` returns a *fresh* session (separate conf/temp views/UDFs, shared cache — `pyspark/sql/session.py:717-735`); `cloneSession()` copies all of that state in, so code relying on `newSession()` isolation silently sees parent state. Notably, Connect's own `SparkSession` deliberately raises `JVM_ATTRIBUTE_NOT_SUPPORTED` for `newSession` (`pyspark/sql/connect/session.py:1019`) — there is no fresh-session construct bound to the same connection, and `cloneSession()` is a developer API with the opposite semantics. Given that, I'd raise `PySparkNotImplementedError` here (consistent with `registerJavaFunction` and `HiveContext`) rather than silently substituting different semantics; `test_newSession_inherits_state` and the Connect leg of `test_newSession_returns_distinct_instance` would flip accordingly, and the PR description's "the
Connect equivalent of `SparkSession.newSession()`" claim should be dropped either way.
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +143,51 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.sql.utils import is_remote
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote():
+ from pyspark.sql.connect import context as _connect_context
+ from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
+
+ session = SparkSession._getActiveSessionOrCreate()
+ # Route to the Connect counterpart so subclasses (e.g. HiveContext) are handled
+ # correctly: the Connect HiveContext._from_session raises PySparkNotImplementedError.
+ connect_cls = getattr(_connect_context, cls.__name__, _connect_context.SQLContext)
Review Comment:
The default-arg fallback silently hands a base Connect `SQLContext` to any user-defined classic subclass: `MyContext.getOrCreate()` in Connect mode returns an object that is not a `MyContext`, and the subclass's attributes vanish with no signal until an `AttributeError`. The classic branch instantiates the actual subclass (`cls._get_or_create` -> `cls(...)`), so this is a behavior divergence, not just a theoretical one. Since the only known subclass is `HiveContext` (already routed by name), raising `PySparkNotImplementedError` when `cls.__name__` has no Connect counterpart would fail loudly instead — one line. Non-blocking given how rare subclassing this 1.x entry point is.
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
Review Comment:
This em-dash is the only non-ASCII character the PR introduces; comments are conventionally ASCII-only.
```suggestion
# Internal module - not part of the public PySpark API surface.
```
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,468 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+# Internal module — not part of the public PySpark API surface.
+# The public SQLContext/HiveContext are in pyspark.sql.context; this module
+# is an implementation detail used by the Connect dispatch in that file.
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 4.3.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 4.3.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+
+ Unlike the classic path (which checks ``_sc._jsc is None`` to detect a dead
+ SparkContext), Connect sessions have no JVM lifecycle sentinel. Instead we
+ re-create whenever the incoming ``sparkSession`` is not the same object as the
+ one stored in the cached context, which handles the case where the previous
+ session was stopped and a new one started.
+ """
+ if (
+ cls._instantiatedContext is None
+ or cls._instantiatedContext.sparkSession is not sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.3.0
+
+ Notes
+ -----
+ Unlike classic :meth:`pyspark.sql.context.SQLContext.newSession`, which returns a fresh
+ session sharing only the table cache, this uses :meth:`SparkSession.cloneSession` and
+ inherits the current session's state.
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.3.0
+
+ .. deprecated:: 4.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 4.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.3.0
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list/tuple of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring the schema.
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.3.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has three columns: ``namespace``, ``tableName`` and
+ ``isTemporary`` (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a
+ table is a temporary one or not).
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ # Use catalog.listTables() rather than SHOW TABLES so the column names are always
+ # (namespace, tableName, isTemporary), matching the classic implementation.
+ # SHOW TABLES returns "database" vs "namespace" depending on the active catalog.
+ rows = [
+ (t.namespace[-1] if t.namespace else "", t.name, t.isTemporary)
Review Comment:
Small parity question: `t.namespace[-1]` keeps only the last namespace part, while classic SHOW TABLES emits the full quoted namespace (`ShowTablesExec.scala:53`, `ident.namespace().quoted`) — so under a v2 catalog with a multi-level namespace the column reads `b` where classic reads `a.b`. Identical for v1 single-part namespaces, which is surely the dominant case for this shim, so non-blocking — but `".".join(t.namespace)` would match classic exactly if you want full parity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306181226
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 4.0.0
+ """
+ return self._from_session(self.sparkSession.cloneSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 4.0.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 4.0.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 4.0.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 4.0.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 4.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ if dbName is None:
+ return self.sparkSession.sql("SHOW TABLES")
+ return self.sparkSession.sql(f"SHOW TABLES IN {dbName}")
Review Comment:
Yes, fix with `catalog.listTables()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3306184477
##########
python/pyspark/sql/context.py:
##########
@@ -142,23 +142,37 @@ def _ssql_ctx(self) -> "JavaObject":
return self._jsqlContext
@classmethod
- def getOrCreate(cls: Type["SQLContext"], sc: "SparkContext") -> "SQLContext":
+ def getOrCreate(cls: Type["SQLContext"], sc: Optional["SparkContext"] = None) -> "SQLContext":
"""
Get the existing SQLContext or create a new one with given SparkContext.
+ When running in Spark Connect mode, ``sc`` is not required and the active
+ :class:`SparkSession` is used automatically.
+
.. versionadded:: 1.6.0
.. deprecated:: 3.0.0
Use :func:`SparkSession.builder.getOrCreate()` instead.
Parameters
----------
- sc : :class:`SparkContext`
+ sc : :class:`SparkContext`, optional
+ Required in classic mode; ignored in Spark Connect mode.
"""
+ from pyspark.util import is_remote_only
+
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
FutureWarning,
)
+ if is_remote_only():
+ from pyspark.sql.connect.context import SQLContext as ConnectSQLContext
+
+ session = SparkSession._getActiveSessionOrCreate()
+ return ConnectSQLContext._get_or_create_from_session( # type: ignore[return-value]
+ session # type: ignore[arg-type]
Review Comment:
Replace with `cast()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4549032564
cc @HyukjinKwon @Yicong-Huang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3150654256
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,451 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ Type,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.connect.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import Row, AtomicType, DataType, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if SQLContext._instantiatedContext is None:
+ SQLContext._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning.
+
+ Used internally by :meth:`newSession` and :meth:`getOrCreate`.
+ """
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def getOrCreate(cls: Type["SQLContext"], sparkSession: "SparkSession") -> "SQLContext":
+ """Get the existing SQLContext or create a new one wrapping the given SparkSession.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ """
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as new session, that has separate SQLConf,
+ registered temporary views and UDFs, but shared table cache.
+
+ .. versionadded:: 1.6.0
+ """
+ return self._from_session(self.sparkSession.newSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property.
+
+ .. versionadded:: 1.3.0
+ """
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] = _NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given key.
+
+ If the key is not set and defaultValue is set, return
+ defaultValue. If the key is not set and defaultValue is not set, return
+ the system default value.
+
+ .. versionadded:: 1.3.0
+ """
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> "UDFRegistration":
+ """Returns a :class:`UDFRegistration` for UDF registration.
+
+ .. versionadded:: 1.3.1
+
+ Returns
+ -------
+ :class:`UDFRegistration`
+ """
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> "UDTFRegistration":
+ """Returns a :class:`UDTFRegistration` for UDTF registration.
+
+ .. versionadded:: 3.5.0
+
+ Returns
+ -------
+ :class:`UDTFRegistration`
+ """
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> DataFrame:
+ """Create a :class:`DataFrame` with single :class:`~pyspark.sql.types.LongType` column
+ named ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
+ step value ``step``.
+
+ .. versionadded:: 1.4.0
+
+ Parameters
+ ----------
+ start : int
+ the start value
+ end : int, optional
+ the end value (exclusive)
+ step : int, optional
+ the incremental step (default: 1)
+ numPartitions : int, optional
+ the number of partitions of the DataFrame
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType] = None
+ ) -> "UserDefinedFunctionLike":
+ """An alias for :func:`spark.udf.register`.
+ See :meth:`pyspark.sql.UDFRegistration.register`.
+
+ .. versionadded:: 1.2.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 2.3.0. Use spark.udf.register instead.", FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] = None
+ ) -> None:
+ """Not supported in Spark Connect.
+
+ .. versionadded:: 2.1.0
+
+ .. deprecated:: 2.3.0
+ Use :func:`spark.udf.registerJavaFunction` instead.
+ """
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Union["pd.DataFrame", "np.ndarray", "pa.Table", Iterable[Any]],
+ schema: Optional[Union[AtomicType, StructType, str, List[str], Tuple[str, ...]]] = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> DataFrame:
+ """Creates a :class:`DataFrame` from an iterable, a :class:`pandas.DataFrame`,
+ or a :class:`pyarrow.Table`.
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ data : iterable
+ an iterable of any kind of SQL data representation (:class:`Row`,
+ :class:`tuple`, ``int``, ``boolean``, etc.), :class:`list`,
+ :class:`pandas.DataFrame`, or :class:`pyarrow.Table`.
+ schema : :class:`~pyspark.sql.types.DataType`, str or list, optional
+ a :class:`~pyspark.sql.types.DataType` or a datatype string or a list of
+ column names.
+ samplingRatio : float, optional
+ the sample ratio of rows used for inferring
+ verifySchema : bool, optional
+ verify data types of every row against schema.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.createDataFrame( # type: ignore[call-overload]
+ data, schema, samplingRatio, verifySchema
+ )
+
+ def registerDataFrameAsTable(self, df: DataFrame, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the catalog.
+
+ Temporary tables exist only during the lifetime of this instance of :class:`SQLContext`.
+
+ .. versionadded:: 1.3.0
+ """
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog.
+
+ .. versionadded:: 1.6.0
+ """
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> DataFrame:
+ """Creates an external table based on the dataset in a data source.
+
+ It returns the DataFrame associated with the external table.
+
+ The data source is specified by the ``source`` and a set of ``options``.
+ If ``source`` is not specified, the default data source configured by
+ ``spark.sql.sources.default`` will be used.
+
+ Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
+ created external table.
+
+ .. versionadded:: 1.3.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> DataFrame:
+ """Returns a :class:`DataFrame` representing the result of the given query.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> DataFrame:
+ """Returns the specified table or view as a :class:`DataFrame`.
+
+ .. versionadded:: 1.0.0
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> DataFrame:
+ """Returns a :class:`DataFrame` containing names of tables in the given database.
+
+ If ``dbName`` is not specified, the current database will be used.
+
+ The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
+ (a column with :class:`~pyspark.sql.types.BooleanType` indicating if a table is a
+ temporary one or not).
+
+ .. versionadded:: 1.3.0
+
+ Parameters
+ ----------
+ dbName: str, optional
+ name of the database to use.
+
+ Returns
+ -------
+ :class:`DataFrame`
+ """
+ listed = self.sparkSession.catalog.listTables(dbName)
+ rows = [
Review Comment:
Why do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55574:
URL: https://github.com/apache/spark/pull/55574#discussion_r3327375272
##########
python/pyspark/sql/connect/context.py:
##########
@@ -0,0 +1,457 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import warnings
+from typing import (
+ Optional,
+ Union,
+ Callable,
+ Any,
+ Iterable,
+ List,
+ Tuple,
+ ClassVar,
+ TYPE_CHECKING,
+)
+
+from pyspark import _NoValue
+from pyspark._globals import _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.dataframe import DataFrame
+from pyspark.sql.connect.readwriter import DataFrameReader
+from pyspark.sql.connect.streaming.readwriter import DataStreamReader
+from pyspark.sql.connect.streaming.query import StreamingQueryManager
+from pyspark.sql.types import AtomicType, BooleanType, DataType, StringType, StructField, StructType
+
+if TYPE_CHECKING:
+ import numpy as np
+ import pandas as pd
+ import pyarrow as pa
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.udf import UDFRegistration
+ from pyspark.sql.connect.udtf import UDTFRegistration
+ from pyspark.sql._typing import UserDefinedFunctionLike
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. However, we are keeping the class
+ here for backward compatibility.
+
+ This is the Spark Connect-compatible implementation. Unlike the classic implementation,
+ it wraps a Connect :class:`SparkSession` directly and does not require a
+ :class:`~pyspark.SparkContext`.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+
+ Parameters
+ ----------
+ sparkSession : :class:`SparkSession`
+ The Connect :class:`SparkSession` to wrap.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: "SparkSession") -> None:
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: "SparkSession") -> "SQLContext":
+ """Return the cached instance or create one from an active Connect SparkSession.
+
+ Called by the classic :meth:`pyspark.sql.context.SQLContext.getOrCreate` when
+ running in Spark Connect mode, so users do not need to import from
+ ``pyspark.sql.connect`` directly.
+ """
+ if cls._instantiatedContext is None:
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new independent server session cloned from this one,
+ with the current session's configuration, temporary views, and registered functions
+ copied into it.
+
+ .. versionadded:: 4.0.0
Review Comment:
Correction in 634fddb1f69 — changed to `4.2.0` (this PR targets the 4.2 release, not 5.0).
---
_This comment was generated with [GitHub MCP](http://go/mcps)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #55574: [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect
URL: https://github.com/apache/spark/pull/55574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57021][CONNECT][PYTHON] Add SQLContext wrapper for Spark Connect [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on PR #55574:
URL: https://github.com/apache/spark/pull/55574#issuecomment-4664567647
The fixes described above are now pushed:
- 9b964998f97 sets `release_session_on_close = True` in both `newSession()` and `cloneSession()` (the latter was a pre-existing copy of the same bug), with regression assertions in `test_connect_context.py` and `test_connect_clone_session.py`.
- 8caf28907d2 overrides `test_newSession_returns_distinct_instance` in `SQLContextParityTests` to release the new server-side session and close its client, fixing the Connect client leak in the parity run.
No code change for item 3 - the "newSession() drops client settings" suggestion was rejected for the reasons above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org