You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/02/25 01:51:34 UTC

[PR] [SPARK-55674][PYTHON] Use pa.RecordBatch.from_pandas for 0-column table in Spark Connect session [spark]

Yicong-Huang opened a new pull request, #54468:
URL: https://github.com/apache/spark/pull/54468

   ### What changes were proposed in this pull request?
   
   Replace `pa.Table.from_struct_array(pa.array([{}] * len(data), type=pa.struct([])))` with `pa.Table.from_batches([pa.RecordBatch.from_pandas(data)])` in `connect/session.py` when handling 0-column pandas DataFrames.
   
   ### Why are the changes needed?
   
   Follow-up to SPARK-55600 (PR #54382), as suggested by @ueshin. The same `pa.RecordBatch.from_pandas` approach used in `pandas/conversion.py` to preserve row count for 0-column DataFrames should also be applied to the equivalent path in `connect/session.py`.
   
   `pa.RecordBatch.from_pandas` preserves `num_rows` via pandas index metadata, which is the same approach used in SPARK-55600.
   
   ### Does this PR introduce any user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Existing tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-55674][PYTHON] Optimize 0-column table conversion in Spark Connect [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #54468: [SPARK-55674][PYTHON] Optimize 0-column table conversion in Spark Connect
URL: https://github.com/apache/spark/pull/54468


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-55674][PYTHON] Optimize 0-column table conversion in Spark Connect [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54468:
URL: https://github.com/apache/spark/pull/54468#issuecomment-3957234816

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-55674][PYTHON] Optimize 0-column table conversion in Spark Connect [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #54468:
URL: https://github.com/apache/spark/pull/54468#issuecomment-3956330865

   > Can't we apply this to ？
   
   Not for this case. This case is "data" is empty but schema is non empty, so we could not use convert and preserve the information from data: the columns will mismatch. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-55674][PYTHON] Optimize 0-column table conversion in Spark Connect [spark]

Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #54468:
URL: https://github.com/apache/spark/pull/54468#issuecomment-3956285960

   Can't we apply this to
   https://github.com/apache/spark/blob/01bfd80852f161c05fdc0cf01b78acee45e005d4/python/pyspark/sql/conversion.py#L289
   ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org