Re: [PR] [SPARK-56286][PYTHON] Add DataFrame.dataQuality API for column profiling [spark]

["HyukjinKwon (via GitHub)" <gi...@apache.org>]

HyukjinKwon commented on code in PR #55095:
URL: https://github.com/apache/spark/pull/55095#discussion_r3037429092


##########
python/pyspark/sql/dataframe.py:
##########
@@ -3380,6 +3380,184 @@ def summary(self, *statistics: str) -> "DataFrame":
         """
         ...
 
+    def dataQuality(self) -> "DataFrame":
+        """Profiles column-level and dataset-level data quality metrics.
+
+        This method returns one row per input column plus one synthetic
+        ``__dataset__`` row containing overall dataset completeness metrics.
+        The resulting :class:`DataFrame` includes null counts, null ratios,
+        distinct counts, min/max, mode, and numeric statistics such as mean,
+        standard deviation, and median.
+
+        Notes
+        -----
+        This function is meant for exploratory data analysis, as we make no
+        guarantee about the backward compatibility of the schema of the resulting
+        :class:`DataFrame`.
+
+        .. versionadded:: 4.1.0
+
+        Returns
+        -------
+        :class:`DataFrame`
+            A new DataFrame that profiles the input DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 10.0, "ok"), (2, None, "ok"), (None, 30.0, None)],
+        ...     ["id", "score", "status"],
+        ... )
+        >>> df.dataQuality().select("column", "null_count", "mean", "median").show()
+        +-----------+----------+----+------+
+        |     column|null_count|mean|median|

Review Comment:
   This really depends on what people think are more important. I would just let other users to decide what to aggregate and show. Doing those all aggregates by default would be extremely expensive, and the basic usages should be enough for describe and summary. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org