[GitHub] [spark] amaliujia commented on a diff in pull request #38803: [SPARK-41114] [CONNECT] [PYTHON] [FOLLOW-UP] Python Client support for local data

[GitBox <gi...@apache.org>]

amaliujia commented on code in PR #38803:
URL: https://github.com/apache/spark/pull/38803#discussion_r1032991360


##########
python/pyspark/sql/connect/plan.py:
##########
@@ -177,6 +178,37 @@ def _repr_html_(self) -> str:
         """
 
 
+class LocalRelation(LogicalPlan):
+    """Creates a LocalRelation plan object based on a Pandas DataFrame."""
+
+    def __init__(self, pdf: "pandas.DataFrame") -> None:
+        super().__init__(None)
+        self._pdf = pdf
+
+    def plan(self, session: "SparkConnectClient") -> proto.Relation:
+        assert self._pdf is not None

Review Comment:
   Nit: is this a bit redundant though that `plan.py` is internal API, the constructor does not accepts Optional pandas dataframe and we have mypy to do type checking



##########
python/pyspark/sql/connect/plan.py:
##########
@@ -177,6 +178,37 @@ def _repr_html_(self) -> str:
         """
 
 
+class LocalRelation(LogicalPlan):
+    """Creates a LocalRelation plan object based on a Pandas DataFrame."""
+
+    def __init__(self, pdf: "pandas.DataFrame") -> None:
+        super().__init__(None)
+        self._pdf = pdf
+
+    def plan(self, session: "SparkConnectClient") -> proto.Relation:
+        assert self._pdf is not None
+
+        sink = pa.BufferOutputStream()
+        table = pa.Table.from_pandas(self._pdf)
+        with pa.ipc.new_stream(sink, table.schema) as writer:

Review Comment:
   I am not familiar with so:
   
   any possible that an empty panda dataframe are used here (e.g. has schema but no data)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org