reading info from spark 2.0 application UI

["TheGeorge1918 ." <zh...@gmail.com>]

Hi all,

I'm deeply confused by the executor configuration in Spark. I have two
machines, each with 40 vcores. By mistake, I assign 7 executors and each
with 11 vcores (It ran without any problem). As a result, one machine has 4
executors and the other has 3 executors + driver. But this means for the
machine with 4 executors, it needs 4 x 11 = 44 vcores which is more than 40
vcores available on that machine. Do I miss something here? Thanks a lot.

aws emr cluster:
2 x m4.10xlarge machine, each with 40 vcores, 160G memory

spark:
num executors: 7
executor memory: 33G
num cores: 11
driver memory: 39G
driver cores: 6

Re: reading info from spark 2.0 application UI

[Sean Owen <so...@cloudera.com>]

What matters in this case is how many vcores YARN thinks it can allocate
per machine. I think the relevant setting is
yarn.nodemanager.resource.cpu-vcores. I bet you'll find this is actually
more than the machine's number of cores, possibly on purpose, to enable
some over-committing.

On Mon, Oct 24, 2016 at 4:13 PM TheGeorge1918 . <zh...@gmail.com>
wrote:

> Yep. I'm pretty sure that 4 executors are on 1 machine. I use yarn in emr.
> I have another "faulty" configuration with 9 executors and 5 of them are on
> one machine. Each one with 9 cores which adds up to 45 cores in that
> machine (the total vcores is 40). Still it works. The total number of
> vcores is 80 in the cluster but I get 81 in total from executors exclusing
> the cores for driver and system. I use aws emr ec2 instance where it
> specifies the resource available for each type of machine. Maybe I could go
> beyond the limitation in the cluster. I just want to make sure I understand
> correctly that when allocating vcores, it means vcores not the threads.
>
> Thanks a lot.
>
> Best
>
>
>
> On Mon, Oct 24, 2016 at 4:55 PM, Sean Owen <so...@cloudera.com> wrote:
>
> If you're really sure that 4 executors are on 1 machine, then it means
> your resource manager allowed it. What are you using, YARN? check that you
> really are limited to 40 cores per machine in the YARN config.
>
> On Mon, Oct 24, 2016 at 3:33 PM TheGeorge1918 . <zh...@gmail.com>
> wrote:
>
> Hi all,
>
> I'm deeply confused by the executor configuration in Spark. I have two
> machines, each with 40 vcores. By mistake, I assign 7 executors and each
> with 11 vcores (It ran without any problem). As a result, one machine has 4
> executors and the other has 3 executors + driver. But this means for the
> machine with 4 executors, it needs 4 x 11 = 44 vcores which is more than 40
> vcores available on that machine. Do I miss something here? Thanks a lot.
>
> aws emr cluster:
> 2 x m4.10xlarge machine, each with 40 vcores, 160G memory
>
> spark:
> num executors: 7
> executor memory: 33G
> num cores: 11
> driver memory: 39G
> driver cores: 6
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>

Re: reading info from spark 2.0 application UI

["TheGeorge1918 ." <zh...@gmail.com>]

Yep. I'm pretty sure that 4 executors are on 1 machine. I use yarn in emr.
I have another "faulty" configuration with 9 executors and 5 of them are on
one machine. Each one with 9 cores which adds up to 45 cores in that
machine (the total vcores is 40). Still it works. The total number of
vcores is 80 in the cluster but I get 81 in total from executors exclusing
the cores for driver and system. I use aws emr ec2 instance where it
specifies the resource available for each type of machine. Maybe I could go
beyond the limitation in the cluster. I just want to make sure I understand
correctly that when allocating vcores, it means vcores not the threads.

Thanks a lot.

Best



On Mon, Oct 24, 2016 at 4:55 PM, Sean Owen <so...@cloudera.com> wrote:

> If you're really sure that 4 executors are on 1 machine, then it means
> your resource manager allowed it. What are you using, YARN? check that you
> really are limited to 40 cores per machine in the YARN config.
>
> On Mon, Oct 24, 2016 at 3:33 PM TheGeorge1918 . <zh...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I'm deeply confused by the executor configuration in Spark. I have two
>> machines, each with 40 vcores. By mistake, I assign 7 executors and each
>> with 11 vcores (It ran without any problem). As a result, one machine has 4
>> executors and the other has 3 executors + driver. But this means for the
>> machine with 4 executors, it needs 4 x 11 = 44 vcores which is more than 40
>> vcores available on that machine. Do I miss something here? Thanks a lot.
>>
>> aws emr cluster:
>> 2 x m4.10xlarge machine, each with 40 vcores, 160G memory
>>
>> spark:
>> num executors: 7
>> executor memory: 33G
>> num cores: 11
>> driver memory: 39G
>> driver cores: 6
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: reading info from spark 2.0 application UI

[Sean Owen <so...@cloudera.com>]

If you're really sure that 4 executors are on 1 machine, then it means your
resource manager allowed it. What are you using, YARN? check that you
really are limited to 40 cores per machine in the YARN config.

On Mon, Oct 24, 2016 at 3:33 PM TheGeorge1918 . <zh...@gmail.com>
wrote:

> Hi all,
>
> I'm deeply confused by the executor configuration in Spark. I have two
> machines, each with 40 vcores. By mistake, I assign 7 executors and each
> with 11 vcores (It ran without any problem). As a result, one machine has 4
> executors and the other has 3 executors + driver. But this means for the
> machine with 4 executors, it needs 4 x 11 = 44 vcores which is more than 40
> vcores available on that machine. Do I miss something here? Thanks a lot.
>
> aws emr cluster:
> 2 x m4.10xlarge machine, each with 40 vcores, 160G memory
>
> spark:
> num executors: 7
> executor memory: 33G
> num cores: 11
> driver memory: 39G
> driver cores: 6
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org