You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Premal Shah <pr...@gmail.com> on 2014/11/11 23:43:41 UTC
(Unknown)
Hi,
We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
balancing to do.
On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
working and were using a ton of network to get the nodes balanced. However,
looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
high I set it to be. The datanodes are using only around 30Mbps. Is there a
way to speed up the balancing?
--
Regards,
Premal Shah.
Re:
Posted by Premal Shah <pr...@gmail.com>.
I'm sorry, I send this email without a subject.
This is from iostat
avg-cpu: %user %nice %system %iowait %steal %idle
1.98 0.00 0.34 0.71 0.02 96.94
When we run MR jobs, I see bandwidth usage spike to 1Gbps easily. But when
there are no jobs running, balancer is not using all the network resources.
On Tue, Nov 11, 2014 at 3:34 PM, daemeon reiydelle <da...@gmail.com>
wrote:
> I observed what I thought was a similar problem but found that I was
> actually hitting the physical disks much harder/more efficiently ... I
> found that I had some spindles with pretty deep queue depths. When I sat
> back and actually measured, I found that my throughput had substantially
> increased, moving the bottleneck to the disks. What does iostats tell you?
>
>
>
>
>
>
>
>
>
> *.......“The race is not to the swift,nor the battle to the strong,but to
> those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
> ReiydelleUSA (+1) 415.501.0198 <%28%2B1%29%20415.501.0198>London (+44) (0)
> 20 8144 9872 <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
> wrote:
>
>> Hi,
>> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
>> balancing to do.
>> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
>> working and were using a ton of network to get the nodes balanced. However,
>> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
>> high I set it to be. The datanodes are using only around 30Mbps. Is there a
>> way to speed up the balancing?
>>
>> --
>> Regards,
>> Premal Shah.
>>
>
>
--
Regards,
Premal Shah.
Re:
Posted by Premal Shah <pr...@gmail.com>.
I'm sorry, I send this email without a subject.
This is from iostat
avg-cpu: %user %nice %system %iowait %steal %idle
1.98 0.00 0.34 0.71 0.02 96.94
When we run MR jobs, I see bandwidth usage spike to 1Gbps easily. But when
there are no jobs running, balancer is not using all the network resources.
On Tue, Nov 11, 2014 at 3:34 PM, daemeon reiydelle <da...@gmail.com>
wrote:
> I observed what I thought was a similar problem but found that I was
> actually hitting the physical disks much harder/more efficiently ... I
> found that I had some spindles with pretty deep queue depths. When I sat
> back and actually measured, I found that my throughput had substantially
> increased, moving the bottleneck to the disks. What does iostats tell you?
>
>
>
>
>
>
>
>
>
> *.......“The race is not to the swift,nor the battle to the strong,but to
> those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
> ReiydelleUSA (+1) 415.501.0198 <%28%2B1%29%20415.501.0198>London (+44) (0)
> 20 8144 9872 <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
> wrote:
>
>> Hi,
>> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
>> balancing to do.
>> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
>> working and were using a ton of network to get the nodes balanced. However,
>> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
>> high I set it to be. The datanodes are using only around 30Mbps. Is there a
>> way to speed up the balancing?
>>
>> --
>> Regards,
>> Premal Shah.
>>
>
>
--
Regards,
Premal Shah.
Re:
Posted by Premal Shah <pr...@gmail.com>.
I'm sorry, I send this email without a subject.
This is from iostat
avg-cpu: %user %nice %system %iowait %steal %idle
1.98 0.00 0.34 0.71 0.02 96.94
When we run MR jobs, I see bandwidth usage spike to 1Gbps easily. But when
there are no jobs running, balancer is not using all the network resources.
On Tue, Nov 11, 2014 at 3:34 PM, daemeon reiydelle <da...@gmail.com>
wrote:
> I observed what I thought was a similar problem but found that I was
> actually hitting the physical disks much harder/more efficiently ... I
> found that I had some spindles with pretty deep queue depths. When I sat
> back and actually measured, I found that my throughput had substantially
> increased, moving the bottleneck to the disks. What does iostats tell you?
>
>
>
>
>
>
>
>
>
> *.......“The race is not to the swift,nor the battle to the strong,but to
> those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
> ReiydelleUSA (+1) 415.501.0198 <%28%2B1%29%20415.501.0198>London (+44) (0)
> 20 8144 9872 <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
> wrote:
>
>> Hi,
>> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
>> balancing to do.
>> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
>> working and were using a ton of network to get the nodes balanced. However,
>> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
>> high I set it to be. The datanodes are using only around 30Mbps. Is there a
>> way to speed up the balancing?
>>
>> --
>> Regards,
>> Premal Shah.
>>
>
>
--
Regards,
Premal Shah.
Re:
Posted by Premal Shah <pr...@gmail.com>.
I'm sorry, I send this email without a subject.
This is from iostat
avg-cpu: %user %nice %system %iowait %steal %idle
1.98 0.00 0.34 0.71 0.02 96.94
When we run MR jobs, I see bandwidth usage spike to 1Gbps easily. But when
there are no jobs running, balancer is not using all the network resources.
On Tue, Nov 11, 2014 at 3:34 PM, daemeon reiydelle <da...@gmail.com>
wrote:
> I observed what I thought was a similar problem but found that I was
> actually hitting the physical disks much harder/more efficiently ... I
> found that I had some spindles with pretty deep queue depths. When I sat
> back and actually measured, I found that my throughput had substantially
> increased, moving the bottleneck to the disks. What does iostats tell you?
>
>
>
>
>
>
>
>
>
> *.......“The race is not to the swift,nor the battle to the strong,but to
> those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
> ReiydelleUSA (+1) 415.501.0198 <%28%2B1%29%20415.501.0198>London (+44) (0)
> 20 8144 9872 <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
> wrote:
>
>> Hi,
>> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
>> balancing to do.
>> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
>> working and were using a ton of network to get the nodes balanced. However,
>> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
>> high I set it to be. The datanodes are using only around 30Mbps. Is there a
>> way to speed up the balancing?
>>
>> --
>> Regards,
>> Premal Shah.
>>
>
>
--
Regards,
Premal Shah.
Re:
Posted by daemeon reiydelle <da...@gmail.com>.
I observed what I thought was a similar problem but found that I was
actually hitting the physical disks much harder/more efficiently ... I
found that I had some spindles with pretty deep queue depths. When I sat
back and actually measured, I found that my throughput had substantially
increased, moving the bottleneck to the disks. What does iostats tell you?
*.......“The race is not to the swift,nor the battle to the strong,but to
those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
wrote:
> Hi,
> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
> balancing to do.
> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
> working and were using a ton of network to get the nodes balanced. However,
> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
> high I set it to be. The datanodes are using only around 30Mbps. Is there a
> way to speed up the balancing?
>
> --
> Regards,
> Premal Shah.
>
Re:
Posted by daemeon reiydelle <da...@gmail.com>.
I observed what I thought was a similar problem but found that I was
actually hitting the physical disks much harder/more efficiently ... I
found that I had some spindles with pretty deep queue depths. When I sat
back and actually measured, I found that my throughput had substantially
increased, moving the bottleneck to the disks. What does iostats tell you?
*.......“The race is not to the swift,nor the battle to the strong,but to
those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
wrote:
> Hi,
> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
> balancing to do.
> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
> working and were using a ton of network to get the nodes balanced. However,
> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
> high I set it to be. The datanodes are using only around 30Mbps. Is there a
> way to speed up the balancing?
>
> --
> Regards,
> Premal Shah.
>
Re:
Posted by daemeon reiydelle <da...@gmail.com>.
I observed what I thought was a similar problem but found that I was
actually hitting the physical disks much harder/more efficiently ... I
found that I had some spindles with pretty deep queue depths. When I sat
back and actually measured, I found that my throughput had substantially
increased, moving the bottleneck to the disks. What does iostats tell you?
*.......“The race is not to the swift,nor the battle to the strong,but to
those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
wrote:
> Hi,
> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
> balancing to do.
> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
> working and were using a ton of network to get the nodes balanced. However,
> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
> high I set it to be. The datanodes are using only around 30Mbps. Is there a
> way to speed up the balancing?
>
> --
> Regards,
> Premal Shah.
>
Re:
Posted by daemeon reiydelle <da...@gmail.com>.
I observed what I thought was a similar problem but found that I was
actually hitting the physical disks much harder/more efficiently ... I
found that I had some spindles with pretty deep queue depths. When I sat
back and actually measured, I found that my throughput had substantially
increased, moving the bottleneck to the disks. What does iostats tell you?
*.......“The race is not to the swift,nor the battle to the strong,but to
those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M.
ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Tue, Nov 11, 2014 at 2:43 PM, Premal Shah <pr...@gmail.com>
wrote:
> Hi,
> We recently upgraded from 1.0.4 to 1.2.1 and now have a ton of DFS
> balancing to do.
> On 1.0.4, when I set the bandwidth to 1 or 10 Gbps, the datanodes got
> working and were using a ton of network to get the nodes balanced. However,
> looks like 1.2.1 is not honoring the setBandWidth setting, no matter how
> high I set it to be. The datanodes are using only around 30Mbps. Is there a
> way to speed up the balancing?
>
> --
> Regards,
> Premal Shah.
>