You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Jack <ja...@gmail.com> on 2013/05/02 12:22:44 UTC
datanode's write_block_op_avg_time value
Hi，
I check the datanode's write_block_op_avg_time on my cluster. It turns
out the value of write_block_op_avg_time is about 20000ms. Is that
normal ? The replication is 3.
Regards,
Jack
Re: datanode's write_block_op_avg_time value
Posted by Harsh J <ha...@cloudera.com>.
What are you finding alarming w.r.t. your cluster?
The metric is simple: When did the write start, and when did it finally
end, for a single block? The difference is the writeBlockOp time. The
average is over a varied collection, which is what you're looking at.
Are your jobs I/O bound? If so, I'd consider that low if the block sizes
are small (64m/128m/256m say). If your jobs or services are CPU/Load bound,
then such a metric makes lesser sense as a block completes only much later
(for example, HBase WALs during no-write periods still keep block write
connection open, bloating up these numbers).
On Thu, May 2, 2013 at 3:52 PM, Jack <ja...@gmail.com> wrote:
> Hi，
> I check the datanode's write_block_op_avg_time on my cluster. It turns out
> the value of write_block_op_avg_time is about 20000ms. Is that normal ? The
> replication is 3.
>
>
>
> Regards,
> Jack
>
--
Harsh J
Re: datanode's write_block_op_avg_time value
Posted by Harsh J <ha...@cloudera.com>.
What are you finding alarming w.r.t. your cluster?
The metric is simple: When did the write start, and when did it finally
end, for a single block? The difference is the writeBlockOp time. The
average is over a varied collection, which is what you're looking at.
Are your jobs I/O bound? If so, I'd consider that low if the block sizes
are small (64m/128m/256m say). If your jobs or services are CPU/Load bound,
then such a metric makes lesser sense as a block completes only much later
(for example, HBase WALs during no-write periods still keep block write
connection open, bloating up these numbers).
On Thu, May 2, 2013 at 3:52 PM, Jack <ja...@gmail.com> wrote:
> Hi，
> I check the datanode's write_block_op_avg_time on my cluster. It turns out
> the value of write_block_op_avg_time is about 20000ms. Is that normal ? The
> replication is 3.
>
>
>
> Regards,
> Jack
>
--
Harsh J
Re: datanode's write_block_op_avg_time value
Posted by Harsh J <ha...@cloudera.com>.
What are you finding alarming w.r.t. your cluster?
The metric is simple: When did the write start, and when did it finally
end, for a single block? The difference is the writeBlockOp time. The
average is over a varied collection, which is what you're looking at.
Are your jobs I/O bound? If so, I'd consider that low if the block sizes
are small (64m/128m/256m say). If your jobs or services are CPU/Load bound,
then such a metric makes lesser sense as a block completes only much later
(for example, HBase WALs during no-write periods still keep block write
connection open, bloating up these numbers).
On Thu, May 2, 2013 at 3:52 PM, Jack <ja...@gmail.com> wrote:
> Hi，
> I check the datanode's write_block_op_avg_time on my cluster. It turns out
> the value of write_block_op_avg_time is about 20000ms. Is that normal ? The
> replication is 3.
>
>
>
> Regards,
> Jack
>
--
Harsh J
Re: datanode's write_block_op_avg_time value
Posted by Harsh J <ha...@cloudera.com>.
What are you finding alarming w.r.t. your cluster?
The metric is simple: When did the write start, and when did it finally
end, for a single block? The difference is the writeBlockOp time. The
average is over a varied collection, which is what you're looking at.
Are your jobs I/O bound? If so, I'd consider that low if the block sizes
are small (64m/128m/256m say). If your jobs or services are CPU/Load bound,
then such a metric makes lesser sense as a block completes only much later
(for example, HBase WALs during no-write periods still keep block write
connection open, bloating up these numbers).
On Thu, May 2, 2013 at 3:52 PM, Jack <ja...@gmail.com> wrote:
> Hi，
> I check the datanode's write_block_op_avg_time on my cluster. It turns out
> the value of write_block_op_avg_time is about 20000ms. Is that normal ? The
> replication is 3.
>
>
>
> Regards,
> Jack
>
--
Harsh J