[GitHub] [spark] tgravescs commented on issue #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance

[GitBox <gi...@apache.org>]

tgravescs commented on issue #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance
URL: https://github.com/apache/spark/pull/26085#issuecomment-541067929
 
 
   I haven't looked at the code yet, can you clarify what ops/ms is measuring here?  
   
   >> For smaller MapStatuses, we find it's 2x faster.
   
   Just to verify, this holds true for small jobs, where you have 10 outputs or 100 outputs?
   
   Did you try with say 2000 or 5000 blocks?  The performance numbers for zstd ops/ms for the 2 step dropped a lot between 5 and 500, whereas the 1 step didn't drop nearly as much, wondering if 1 step becomes faster at some point.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org