You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by 柳松 <la...@126.com> on 2009/03/06 05:26:01 UTC
The cpu preemption between MPI and Hadoop programs on Same Cluster
Dear all:
I run my hadoop program with another MPI program on the same cluster. here is the result of "top".
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11750 qianglv 25 0 233m 99m 6100 R 99.7 2.5 116:05.59 rosetta.mpich
18094 cip 17 0 3136m 68m 15m S 0.5 1.7 0:12.69 java
18244 cip 17 0 3142m 80m 15m S 0.2 2.0 0:17.61 java
18367 cip 18 0 2169m 88m 15m S 0.1 2.3 0:17.46 java
18012 cip 18 0 3141m 77m 15m S 0.1 2.0 0:14.49 java
18584 cip 21 0 1111m 46m 15m S 0.1 1.2 0:05.12 java
My Hadoop program can only get no more than 1 percent cpu time slide in total, compared with the rosetta.mpich program's 99.7%.
I'm sure my program is in progress since the log files told me, they are running normally.
Someone told me, it's the nature of Java program, low cpu priority, especially compared with C program.
Is that true?
Regards
Song Liu in Suzhou University.
Re: The cpu preemption between MPI and Hadoop programs on Same
Cluster
Posted by Aaron Kimball <aa...@cloudera.com>.
Song, you should be able to use 'nice' to reprioritize the MPI task
below that of your Hadoop jobs.
- Aaron
On Thu, Mar 5, 2009 at 8:26 PM, 柳松 <la...@126.com> wrote:
>
> Dear all:
> I run my hadoop program with another MPI program on the same cluster. here is the result of "top".
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 11750 qianglv 25 0 233m 99m 6100 R 99.7 2.5 116:05.59 rosetta.mpich
> 18094 cip 17 0 3136m 68m 15m S 0.5 1.7 0:12.69 java
> 18244 cip 17 0 3142m 80m 15m S 0.2 2.0 0:17.61 java
> 18367 cip 18 0 2169m 88m 15m S 0.1 2.3 0:17.46 java
> 18012 cip 18 0 3141m 77m 15m S 0.1 2.0 0:14.49 java
> 18584 cip 21 0 1111m 46m 15m S 0.1 1.2 0:05.12 java
>
> My Hadoop program can only get no more than 1 percent cpu time slide in total, compared with the rosetta.mpich program's 99.7%.
>
> I'm sure my program is in progress since the log files told me, they are running normally.
>
> Someone told me, it's the nature of Java program, low cpu priority, especially compared with C program.
>
> Is that true?
>
> Regards
> Song Liu in Suzhou University.