You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jelez Raditchkov <je...@hotmail.com> on 2016/03/04 22:09:59 UTC

Best way to merge files from streaming jobs

My streaming job is creating files on S3.The problem is that those files end up very small if I just write them to S3 directly.This is why I use coalesce() to reduce the number of files and make them larger.
However, coalesce shuffles data and my job processing time ends up higher than sparkBatchIntervalMilliseconds.
I have observed that if I coalesce the number of partitions to be equal to the cores in the cluster I get less shuffling - but that is unsubstantiated.Is there any dependency/rule between number of executors, number of cores etc. that I can use to minimize shuffling and at the same time achieve minimum number of output files per batch?What is the best practice?