You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alex Dzhagriev <dz...@gmail.com> on 2016/02/23 15:49:00 UTC
reasonable number of executors
Hello all,
Can someone please advise me on the pros and cons on how to allocate the
resources: many small heap machines with 1 core or few machines with big
heaps and many cores? I'm sure that depends on the data flow and there is
no best practise solution. E.g. with bigger heap I can perform map-side
join with bigger table. What other considerations should I keep in mind in
order to choose the right configuration?
Thanks, Alex.
Re: reasonable number of executors
Posted by Alex Dzhagriev <dz...@gmail.com>.
Hi Igor,
That's a great talk and an exact answer to my question. Thank you.
Cheers, Alex.
On Tue, Feb 23, 2016 at 8:27 PM, Igor Berman <ig...@gmail.com> wrote:
>
> http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications
>
> there is a section that is connected to your question
>
> On 23 February 2016 at 16:49, Alex Dzhagriev <dz...@gmail.com> wrote:
>
>> Hello all,
>>
>> Can someone please advise me on the pros and cons on how to allocate the
>> resources: many small heap machines with 1 core or few machines with big
>> heaps and many cores? I'm sure that depends on the data flow and there is
>> no best practise solution. E.g. with bigger heap I can perform map-side
>> join with bigger table. What other considerations should I keep in mind in
>> order to choose the right configuration?
>>
>> Thanks, Alex.
>>
>
>
Re: reasonable number of executors
Posted by Igor Berman <ig...@gmail.com>.
http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications
there is a section that is connected to your question
On 23 February 2016 at 16:49, Alex Dzhagriev <dz...@gmail.com> wrote:
> Hello all,
>
> Can someone please advise me on the pros and cons on how to allocate the
> resources: many small heap machines with 1 core or few machines with big
> heaps and many cores? I'm sure that depends on the data flow and there is
> no best practise solution. E.g. with bigger heap I can perform map-side
> join with bigger table. What other considerations should I keep in mind in
> order to choose the right configuration?
>
> Thanks, Alex.
>
Re: reasonable number of executors
Posted by Jorge Machado <jo...@me.com>.
Hi Alex,
take a look here :
https://blogs.aws.amazon.com/bigdata/post/Tx3RD6EISZGHQ1C/The-Impact-of-Using-Latest-Generation-Instances-for-Your-Amazon-EMR-Job <https://blogs.aws.amazon.com/bigdata/post/Tx3RD6EISZGHQ1C/The-Impact-of-Using-Latest-Generation-Instances-for-Your-Amazon-EMR-Job>
Basically it depends of your type of workload. Will you need Cache ?
Jorge Machado
www.jmachado.me
> On 23/02/2016, at 15:49, Alex Dzhagriev <dz...@gmail.com> wrote:
>
> Hello all,
>
> Can someone please advise me on the pros and cons on how to allocate the resources: many small heap machines with 1 core or few machines with big heaps and many cores? I'm sure that depends on the data flow and there is no best practise solution. E.g. with bigger heap I can perform map-side join with bigger table. What other considerations should I keep in mind in order to choose the right configuration?
>
> Thanks, Alex.