You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/05/21 05:29:24 UTC
[GitHub] [spark] BryanCutler commented on issue #24650:
[SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow
enabled
BryanCutler commented on issue #24650: [SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow enabled
URL: https://github.com/apache/spark/pull/24650#issuecomment-494245140
I had another thought about this, the stuff in `doAfterLastPartition` could be removed from `handlePartitionBatches` and called after `runJob` regardless if partitions are empty or not. This really wouldn't make any difference performance-wise because it's just moving it outside the callback function and Python is waiting on it anyway.
It also has the benefit that the number of partitions would not have to be kept track of, so the variables `partitionCount` and `numPartitions` could be removed. It would be a lot clearer then too.
What do you think @dvogelbacher and @HyukjinKwon ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org