You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Himanish Kushary <hi...@gmail.com> on 2012/08/23 21:14:00 UTC
I/O stats interpretation during concurrent hive M/R runs
Hi,
I am curious about interpretation of the output from iostat on a datanode
during a M/R run.I want to understand how to diagnosis a disk i/o issue in
a hadoop cluster.
Is there any good documentation to help me understand the results from
iostats in Hadoop context ?
Here are the iostat output from a DN while two intensive M/R jobs were
executing.Does this result indicate any performance issue related to the
disks ?
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.80 0.00 59.20 0 296
sdb 1436.20 96376.00 211424.00 481880 1057120
avg-cpu: %user %nice %system %iowait %steal %idle
22.81 0.00 45.28 3.59 0.00 28.32
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 2.80 25.60 80.00 128 400
sdb 1073.60 45891.20 203473.60 229456 1017368
avg-cpu: %user %nice %system %iowait %steal %idle
17.28 0.00 74.49 0.32 0.00 7.92
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 3.44 0.00 83.97 0 440
sdb 1174.62 52370.99 209789.31 274424 1099296
avg-cpu: %user %nice %system %iowait %steal %idle
51.72 0.00 47.60 0.31 0.00 0.38
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.20 0.00 22.40 0 112
sdb 1094.20 67492.80 177187.20 337464 885936
avg-cpu: %user %nice %system %iowait %steal %idle
43.73 0.00 36.19 3.03 0.00 17.05
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.80 0.00 46.40 0 232
sdb 1241.20 100969.60 162806.40 504848 814032
avg-cpu: %user %nice %system %iowait %steal %idle
37.09 0.00 58.61 0.77 0.00 3.54
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 4.60 182.40 19.20 912 96
sdb 1235.20 47780.80 235912.00 238904 1179560
avg-cpu: %user %nice %system %iowait %steal %idle
47.23 0.00 42.49 3.09 0.00 7.19
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.60 0.00 46.40 0 232
sdb 1005.20 86502.40 135886.40 432512 679432
---------------------------
Thanks & Regards
Himanish
Re: I/O stats interpretation during concurrent hive M/R runs
Posted by Himanish Kushary <hi...@gmail.com>.
After sending this message I issued the iostat -dxm 5 command on the
DNs.... the %util column shows 70-80 average value sometimes going up to
90-100 for few seconds
Does this mean the disk is becoming the bottleneck ? or is this normal ?
On Thu, Aug 23, 2012 at 3:14 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Hi,
>
> I am curious about interpretation of the output from iostat on a datanode
> during a M/R run.I want to understand how to diagnosis a disk i/o issue in
> a hadoop cluster.
>
> Is there any good documentation to help me understand the results from
> iostats in Hadoop context ?
>
> Here are the iostat output from a DN while two intensive M/R jobs were
> executing.Does this result indicate any performance issue related to the
> disks ?
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 59.20 0 296
> sdb 1436.20 96376.00 211424.00 481880 1057120
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 22.81 0.00 45.28 3.59 0.00 28.32
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 2.80 25.60 80.00 128 400
> sdb 1073.60 45891.20 203473.60 229456 1017368
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 17.28 0.00 74.49 0.32 0.00 7.92
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 3.44 0.00 83.97 0 440
> sdb 1174.62 52370.99 209789.31 274424 1099296
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 51.72 0.00 47.60 0.31 0.00 0.38
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.20 0.00 22.40 0 112
> sdb 1094.20 67492.80 177187.20 337464 885936
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 43.73 0.00 36.19 3.03 0.00 17.05
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 46.40 0 232
> sdb 1241.20 100969.60 162806.40 504848 814032
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 37.09 0.00 58.61 0.77 0.00 3.54
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 4.60 182.40 19.20 912 96
> sdb 1235.20 47780.80 235912.00 238904 1179560
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 47.23 0.00 42.49 3.09 0.00 7.19
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.60 0.00 46.40 0 232
> sdb 1005.20 86502.40 135886.40 432512 679432
>
>
>
> ---------------------------
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish
Re: I/O stats interpretation during concurrent hive M/R runs
Posted by Himanish Kushary <hi...@gmail.com>.
After sending this message I issued the iostat -dxm 5 command on the
DNs.... the %util column shows 70-80 average value sometimes going up to
90-100 for few seconds
Does this mean the disk is becoming the bottleneck ? or is this normal ?
On Thu, Aug 23, 2012 at 3:14 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Hi,
>
> I am curious about interpretation of the output from iostat on a datanode
> during a M/R run.I want to understand how to diagnosis a disk i/o issue in
> a hadoop cluster.
>
> Is there any good documentation to help me understand the results from
> iostats in Hadoop context ?
>
> Here are the iostat output from a DN while two intensive M/R jobs were
> executing.Does this result indicate any performance issue related to the
> disks ?
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 59.20 0 296
> sdb 1436.20 96376.00 211424.00 481880 1057120
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 22.81 0.00 45.28 3.59 0.00 28.32
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 2.80 25.60 80.00 128 400
> sdb 1073.60 45891.20 203473.60 229456 1017368
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 17.28 0.00 74.49 0.32 0.00 7.92
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 3.44 0.00 83.97 0 440
> sdb 1174.62 52370.99 209789.31 274424 1099296
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 51.72 0.00 47.60 0.31 0.00 0.38
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.20 0.00 22.40 0 112
> sdb 1094.20 67492.80 177187.20 337464 885936
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 43.73 0.00 36.19 3.03 0.00 17.05
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 46.40 0 232
> sdb 1241.20 100969.60 162806.40 504848 814032
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 37.09 0.00 58.61 0.77 0.00 3.54
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 4.60 182.40 19.20 912 96
> sdb 1235.20 47780.80 235912.00 238904 1179560
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 47.23 0.00 42.49 3.09 0.00 7.19
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.60 0.00 46.40 0 232
> sdb 1005.20 86502.40 135886.40 432512 679432
>
>
>
> ---------------------------
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish
Re: I/O stats interpretation during concurrent hive M/R runs
Posted by Himanish Kushary <hi...@gmail.com>.
After sending this message I issued the iostat -dxm 5 command on the
DNs.... the %util column shows 70-80 average value sometimes going up to
90-100 for few seconds
Does this mean the disk is becoming the bottleneck ? or is this normal ?
On Thu, Aug 23, 2012 at 3:14 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Hi,
>
> I am curious about interpretation of the output from iostat on a datanode
> during a M/R run.I want to understand how to diagnosis a disk i/o issue in
> a hadoop cluster.
>
> Is there any good documentation to help me understand the results from
> iostats in Hadoop context ?
>
> Here are the iostat output from a DN while two intensive M/R jobs were
> executing.Does this result indicate any performance issue related to the
> disks ?
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 59.20 0 296
> sdb 1436.20 96376.00 211424.00 481880 1057120
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 22.81 0.00 45.28 3.59 0.00 28.32
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 2.80 25.60 80.00 128 400
> sdb 1073.60 45891.20 203473.60 229456 1017368
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 17.28 0.00 74.49 0.32 0.00 7.92
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 3.44 0.00 83.97 0 440
> sdb 1174.62 52370.99 209789.31 274424 1099296
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 51.72 0.00 47.60 0.31 0.00 0.38
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.20 0.00 22.40 0 112
> sdb 1094.20 67492.80 177187.20 337464 885936
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 43.73 0.00 36.19 3.03 0.00 17.05
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 46.40 0 232
> sdb 1241.20 100969.60 162806.40 504848 814032
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 37.09 0.00 58.61 0.77 0.00 3.54
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 4.60 182.40 19.20 912 96
> sdb 1235.20 47780.80 235912.00 238904 1179560
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 47.23 0.00 42.49 3.09 0.00 7.19
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.60 0.00 46.40 0 232
> sdb 1005.20 86502.40 135886.40 432512 679432
>
>
>
> ---------------------------
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish
Re: I/O stats interpretation during concurrent hive M/R runs
Posted by Himanish Kushary <hi...@gmail.com>.
After sending this message I issued the iostat -dxm 5 command on the
DNs.... the %util column shows 70-80 average value sometimes going up to
90-100 for few seconds
Does this mean the disk is becoming the bottleneck ? or is this normal ?
On Thu, Aug 23, 2012 at 3:14 PM, Himanish Kushary <hi...@gmail.com>wrote:
> Hi,
>
> I am curious about interpretation of the output from iostat on a datanode
> during a M/R run.I want to understand how to diagnosis a disk i/o issue in
> a hadoop cluster.
>
> Is there any good documentation to help me understand the results from
> iostats in Hadoop context ?
>
> Here are the iostat output from a DN while two intensive M/R jobs were
> executing.Does this result indicate any performance issue related to the
> disks ?
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 59.20 0 296
> sdb 1436.20 96376.00 211424.00 481880 1057120
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 22.81 0.00 45.28 3.59 0.00 28.32
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 2.80 25.60 80.00 128 400
> sdb 1073.60 45891.20 203473.60 229456 1017368
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 17.28 0.00 74.49 0.32 0.00 7.92
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 3.44 0.00 83.97 0 440
> sdb 1174.62 52370.99 209789.31 274424 1099296
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 51.72 0.00 47.60 0.31 0.00 0.38
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.20 0.00 22.40 0 112
> sdb 1094.20 67492.80 177187.20 337464 885936
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 43.73 0.00 36.19 3.03 0.00 17.05
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.80 0.00 46.40 0 232
> sdb 1241.20 100969.60 162806.40 504848 814032
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 37.09 0.00 58.61 0.77 0.00 3.54
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 4.60 182.40 19.20 912 96
> sdb 1235.20 47780.80 235912.00 238904 1179560
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 47.23 0.00 42.49 3.09 0.00 7.19
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 1.60 0.00 46.40 0 232
> sdb 1005.20 86502.40 135886.40 432512 679432
>
>
>
> ---------------------------
> Thanks & Regards
> Himanish
>
--
Thanks & Regards
Himanish