[GitHub] [spark] lidavidm commented on pull request #29818: [SPARK-32953][PYTHON] Add Arrow self_destruct support to toPandas

[GitBox <gi...@apache.org>]

lidavidm commented on pull request #29818:
URL: https://github.com/apache/spark/pull/29818#issuecomment-710181028


   I can't reply to this inline somehow, but
   
   > This looks a little strange to me too. Is concat_arrays actually doing anything here, and if so, wouldn't it do it for the case that selfDestruct is False too?
   
   1) Yes, it's copying each column in the record batch into its own allocation, as explained in the latest commit. I couldn't find a more explicit way in Arrow to copy a column out of a record batch.
   2) Yes, I can try to thread through a parameter, though I think it's harmless in this case.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org