[GitHub] [spark] WeichenXu123 commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)

[GitBox <gi...@apache.org>]

WeichenXu123 commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
URL: https://github.com/apache/spark/pull/24734#issuecomment-497297633
 
 
   @felixcheung The performance matters in the case batch-size small and UDF do heavy computation.
   Suppose two UDF get pipelined, each UDF consume 3s on each batch (and suppose the worker node have enough cores to parallely run the 2 UDFs). And suppose the buffer can hold 100 batches. Then:
   
   My PR:  The first 100 batches computation will consume time about: 100 * 3s.
   Master code: The first 100 batches computation will consume time about 100 * (3 + 3) s.
   
   That's the difference. Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org