Re: Number of Executors per worker process

[Spico Florin <sp...@gmail.com>]

Hello!
  Thank you very much for your response. In the book "Learning Spark" I
found out the following sentence:

"Each application will have at most one executor on each worker"

So worker can have one or none executor process spawned (perhaps the number
depends on the workload distribution).


Best regards,

 Florin

On Thu, Feb 26, 2015 at 1:04 PM, Jeffrey Jedele <je...@gmail.com>
wrote:

> Hi Spico,
>
> Yes, I think an "executor core" in Spark is basically a thread in a worker
> pool. It's recommended to have one executor core per physical core on your
> machine for best performance, but I think in theory you can create as many
> threads as your OS allows.
>
> For deployment:
> There seems to be the actual worker JVM which coordinates the work on a
> worker node. I don't think the actual thread pool lives in there, but a
> separate JVM is created for each application that has cores allocated on
> the node. Otherwise it would be rather hard to impose memory limits on
> application level and it would have serious disadvantages regarding
> stability.
>
> You can check this behavior by looing at the processes on your machine:
> ps aux | grep spark.deploy => will show  master, worker (coordinator) and
> driver JVMs
> ps aux | grep spark.executor => will show the actual worker JVMs
>
> 2015-02-25 14:23 GMT+01:00 Spico Florin <sp...@gmail.com>:
>
>> Hello!
>>  I've read the documentation about the spark architecture, I have the
>> following questions:
>> 1: how many executors can be on a single worker process (JMV)?
>> 2:Should I think executor like a Java Thread Executor where the pool size
>> is equal with the number of the given cores (set up by the
>> SPARK_WORKER_CORES)?
>> 3. If the worker can have many executors, how this is handled by the
>> Spark? Or can I handle by myself to set up the number of executors per JVM?
>>
>> I look forward for your answers.
>>   Regards,
>>   Florin
>>
>
>