Writing to HDFS and cluster utilization

[Alessandro Liparoti <al...@gmail.com>]

Hi,

I would like to briefly present you my use case and gather possible useful
suggestions from the community. I am developing a spark job which massively
read from and write to Hive. Usually, I use 200 executors with 12g memory
each and a parallelism level of 600. The main run of the application
consists of phases: read from hdfs, persist, small and simple aggregations,
write to hdfs. These steps are repeated a certain number of time.
When I write to Hive, I aim to have partitions of approximately 50/70mb,
therefore I repartition before writing in output in approximately 15 parts
(according to the data size). The writing phase takes around 1.5 minutes;
this means that for 1.5 minutes only 15 out of 600 possible active tasks
are running in parallel. This looks a big waste of resources. How would you
solve the problem?

I am trying to experiment with the FAIR scheduler and job pools, but it
seems not improving a lot; for some reasons, I cannot have more than 4
parallel jobs running. I am investigating this opportunity right now, maybe
I will provide more details about it afterwards.

I would like to know if this use case is normal, what would you do and if
in your opinion I am doing something wrong.

Thanks,
*Alessandro Liparoti*

Re: Writing to HDFS and cluster utilization

[Rohit Karlupia <ro...@qubole.com>]

Hi,


The minimal solution is to enable dynamicAllocation and set idle timeout to
low value. This will ensure that idle executors are killed and resources
available for others to use,


spark.dynamicAllocation.enabled

spark.dynamicAllocation.executorIdleTimeout

If you would like to understand "wastage" vs "completion time"  with
different executor counts, try Sparklens https://github.com/qubole/sparklens

thanks,
rohitk








On Fri, Jun 15, 2018 at 6:32 PM, Alessandro Liparoti <
alessandro.lipa@gmail.com> wrote:

> Hi,
>
> I would like to briefly present you my use case and gather possible useful
> suggestions from the community. I am developing a spark job which massively
> read from and write to Hive. Usually, I use 200 executors with 12g memory
> each and a parallelism level of 600. The main run of the application
> consists of phases: read from hdfs, persist, small and simple aggregations,
> write to hdfs. These steps are repeated a certain number of time.
> When I write to Hive, I aim to have partitions of approximately 50/70mb,
> therefore I repartition before writing in output in approximately 15 parts
> (according to the data size). The writing phase takes around 1.5 minutes;
> this means that for 1.5 minutes only 15 out of 600 possible active tasks
> are running in parallel. This looks a big waste of resources. How would you
> solve the problem?
>
> I am trying to experiment with the FAIR scheduler and job pools, but it
> seems not improving a lot; for some reasons, I cannot have more than 4
> parallel jobs running. I am investigating this opportunity right now, maybe
> I will provide more details about it afterwards.
>
> I would like to know if this use case is normal, what would you do and if
> in your opinion I am doing something wrong.
>
> Thanks,
> *Alessandro Liparoti*
>