[GitHub] [spark] zhengruifeng commented on a diff in pull request #37233: [SPARK-39823][PYTHON] Add DataFrame.asSchema(StructType) in PySpark

[GitBox <gi...@apache.org>]

zhengruifeng commented on code in PR #37233:
URL: https://github.com/apache/spark/pull/37233#discussion_r926211928


##########
python/pyspark/sql/dataframe.py:
##########
@@ -1422,6 +1422,61 @@ def colRegex(self, colName: str) -> Column:
         jc = self._jdf.colRegex(colName)
         return Column(jc)
 
+    def asSchema(self, schema: StructType) -> "DataFrame":
+        """
+        Returns a new :class:`DataFrame` where each row is reconciled to match the specified
+        schema.
+
+        Spark will:
+
+        1, Reorder columns and/or inner fields by name to match the specified schema.
+
+        2, Project away columns and/or inner fields that are not needed by the specified schema.
+        Missing columns and/or inner fields (present in the specified schema but not input
+        DataFrame) lead to failures.
+
+        3, Cast the columns and/or inner fields to match the data types in the specified schema,
+        if the types are compatible, e.g., numeric to numeric (error if overflows), but not string
+        to int.
+
+        4, Carry over the metadata from the specified schema, while the columns and/or inner fields
+        still keep their own metadata if not overwritten by the specified schema.
+
+        5, Fail if the nullability are not compatible. For example, the column and/or inner field
+        is nullable but the specified schema requires them to be not nullable.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        schema : :class:`StructType`
+            Specified schema.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame([("a", 1)], ["i", "j"])
+        >>> df.schema
+        StructType([StructField('i', StringType(), True), StructField('j', LongType(), True)])
+        >>> schema = StructType([StructField("j", StringType()), StructField("i", StringType())])
+        >>> df2 = df.asSchema(schema)
+        >>> df2.schema
+        StructType([StructField('j', StringType(), True), StructField('i', StringType(), True)])
+        >>> df2.show()
+        +---+---+
+        |  j|  i|
+        +---+---+
+        |  1|  a|
+        +---+---+
+        """
+        assert schema is not None
+        sc = self.sparkSession._sc
+        assert sc is not None and sc._jvm is not None
+        _struct_type = getattr(
+            getattr(sc._jvm.org.apache.spark.sql.types, "StructType$"), "MODULE$"
+        )
+        jschema = _struct_type.fromString(schema.json())

Review Comment:
   I think it is not very common, but still used in several places, such as: https://github.com/apache/spark/blob/master/python/pyspark/sql/session.py#L324-L327
   
   Here need to use it, since `fromString` is `private[sql]`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org