You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2023/03/02 00:14:08 UTC
[spark] branch branch-3.4 updated: [SPARK-42458][CONNECT][PYTHON] Fixes createDataFrame to support DDL string as schema
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new aefebe224da [SPARK-42458][CONNECT][PYTHON] Fixes createDataFrame to support DDL string as schema
aefebe224da is described below
commit aefebe224da63deb9dfb6ea32bf27cd372ecd84a
Author: Takuya UESHIN <ue...@databricks.com>
AuthorDate: Thu Mar 2 09:13:46 2023 +0900
[SPARK-42458][CONNECT][PYTHON] Fixes createDataFrame to support DDL string as schema
### What changes were proposed in this pull request?
Fixes `createDataFrame` to support DDL string as schema.
### Why are the changes needed?
Currently DDL string as schema is ignored when the data is Python objects and the inference fails:
```py
>>> spark.createDataFrame([(100, None)], "age INT, name STRING").show()
Traceback (most recent call last):
...
ValueError: Some of types cannot be determined after inferring, a StructType Schema is required in this case
```
### Does this PR introduce _any_ user-facing change?
The DDL string as schema will not be ignored when the schema can't be inferred from the given data.
```py
>>> spark.createDataFrame([(100, None)], "age INT, name STRING").show()
+---+----+
|age|name|
+---+----+
|100|null|
+---+----+
```
### How was this patch tested?
Enabled related tests.
Closes #40240 from ueshin/issues/SPARK-42458/schema_str.
Authored-by: Takuya UESHIN <ue...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 4ccc88c49f330d366d15b6e9abeee6e97504006f)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/sql/connect/readwriter.py | 3 ---
python/pyspark/sql/connect/session.py | 2 ++
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/connect/readwriter.py b/python/pyspark/sql/connect/readwriter.py
index 9c9c79cb6eb..d20eaa44148 100644
--- a/python/pyspark/sql/connect/readwriter.py
+++ b/python/pyspark/sql/connect/readwriter.py
@@ -692,9 +692,6 @@ def _test() -> None:
globs = pyspark.sql.connect.readwriter.__dict__.copy()
- # TODO(SPARK-42458): createDataFrame should support DDL string as schema
- del pyspark.sql.connect.readwriter.DataFrameWriter.option.__doc__
-
globs["spark"] = (
PySparkSession.builder.appName("sql.connect.readwriter tests")
.remote("local[4]")
diff --git a/python/pyspark/sql/connect/session.py b/python/pyspark/sql/connect/session.py
index 6b501b7c375..04e4f9e88d0 100644
--- a/python/pyspark/sql/connect/session.py
+++ b/python/pyspark/sql/connect/session.py
@@ -313,6 +313,8 @@ class SparkSession:
# For cases like createDataFrame([("Alice", None, 80.1)], schema)
# we can not infer the schema from the data itself.
warnings.warn("failed to infer the schema from data")
+ if _schema is None and _schema_str is not None:
+ _schema = self.createDataFrame([], schema=_schema_str).schema
if _schema is None or not isinstance(_schema, StructType):
raise ValueError(
"Some of types cannot be determined after inferring, "
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org