CPU Parallelization not being used (local mode)

[Sa...@wellsfargo.com]

Hi all,

would like some insight. I am currently computing huge databases, and playing with monitoring and tunning.

When monitoring the multiple cores I have, I see that even when RDDs are parallelized, computation on the RDD jump from core to core sporadically ( I guess, depending on where the chunk is), So I see one CORE at 100% usage, and the other ones sitting idle by, after some time when the task is complete, the procesing jumps into another core, and so on.

can you share any general insight on this situation? Does this depend on the computation?
I have tried serialization and different setups, but I neve see more than 1 Core working at a spark-submission.

note: This is no cluster mode, just local processors.

Thanks,
Saif

Re: CPU Parallelization not being used (local mode)

[Akhil Das <ak...@sigmoidanalytics.com>]

When you give master as *local[1]* it occupies a single thread which will
probably run on a single core, you can give *local[*]* to allocate the
total # cores that you have.

Thanks
Best Regards

On Tue, Jul 28, 2015 at 1:27 AM, <Sa...@wellsfargo.com> wrote:

>  Hi all,
>
> would like some insight. I am currently computing huge databases, and
> playing with monitoring and tunning.
>
> When monitoring the multiple cores I have, I see that even when RDDs are
> parallelized, computation on the RDD jump from core to core sporadically (
> I guess, depending on where the chunk is), So I see one CORE at 100% usage,
> and the other ones sitting idle by, after some time when the task is
> complete, the procesing jumps into another core, and so on.
>
> can you share any general insight on this situation? Does this depend on
> the computation?
> I have tried serialization and different setups, but I neve see more than
> 1 Core working at a spark-submission.
>
> note: This is no cluster mode, just local processors.
>
> Thanks,
> Saif
>
>