You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kazuki Ohta <ka...@gmail.com> on 2011/05/19 06:02:41 UTC
Re: massive zk expirations under heavy network load
Hi, all
Finally got my cluster stable by upgrading network driver, instead of
changing the
switch.
We used e1000e driver on CentOS 5.5, then upgrading to e1000e-1.3.10a.tar.gz,
the most recent one. This dramatically reduces the # of dropped packets under
heavy load.
> http://bit.ly/lTJsV1
Thanks for the help!
Kazuki
On Thu, Apr 21, 2011 at 11:24 AM, Kazuki Ohta <ka...@gmail.com> wrote:
> Hi, All
>
> Thanks for the helpful comments!
> Nice to see this happens rarely in other environments.
>
> Actually I've changed the configuration not to run the task on the master node,
> but the same problem happened.
>
> So at first, upgrade the switch. Report again if the problem will be fixed.
>
> Thanks
> Kazuki
>
> On Thu, Apr 21, 2011 at 5:32 AM, Gary Helmling <gh...@gmail.com> wrote:
>>> I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15).
>>> The configuraiton is below.
>>>
>>> hdp0: zk + master + region + nn + dn + jt + tt
>>> hdp1: zk + master + region + snn + dn + tt
>>> hdp2: zk + region + dn + tt
>>> hdp3 to hdp15: region + dn + tt
>>>
>>>
>> I would also look at the memory configuration for your servers and the
>> amount of heap allocated to each process. Is it possible hdp0 is swapping
>> when running a MR job? Swapping will cause big headaches and is often a
>> culprit for zk session timeouts.
>>
>> Between the 7 processes it has plus any child tasks started, it's not hard
>> to picture overcommitting memory.
>>
>> Regardless of whether the core problem lies in network hardware or here, I
>> would remove the region server, data node, and task tracker processes from
>> hdp0 and hdp1 for smoother operation.
>>
>> --gh
>>
>
>
>
> --
> --------------------------------------------------
> Kazuki Ohta: http://kzk9.net/
> CTO at Preferred Infrastructure: http://preferred.jp/
>
--
--------------------------------------------------
Kazuki Ohta: http://kzk9.net/
CTO at Preferred Infrastructure: http://preferred.jp/
Re: massive zk expirations under heavy network load
Posted by Kazuki Ohta <ka...@gmail.com>.
Hi, Todd
Thx for replying in Japanese :-) lspci shows the following hardware.
$ lspci | grep Ether
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
> http://ark.intel.com/Product.aspx?id=32209
Regards
Kazuki
2011/5/19 Todd Lipcon <to...@cloudera.com>:
> こんいちは大田さん ：） (still practicing Japanese over here)
>
> Could you paste the output from "lspci" to the list? It would be
> useful to know which particular hardware you had this problem with, so
> we can watch out for it and know to upgrade e1000
>
> Thanks
> -Todd
>
> On Wed, May 18, 2011 at 9:02 PM, Kazuki Ohta <ka...@gmail.com> wrote:
>> Hi, all
>>
>> Finally got my cluster stable by upgrading network driver, instead of
>> changing the
>> switch.
>>
>> We used e1000e driver on CentOS 5.5, then upgrading to e1000e-1.3.10a.tar.gz,
>> the most recent one. This dramatically reduces the # of dropped packets under
>> heavy load.
>>
>>> http://bit.ly/lTJsV1
>>
>> Thanks for the help!
>> Kazuki
>>
>> On Thu, Apr 21, 2011 at 11:24 AM, Kazuki Ohta <ka...@gmail.com> wrote:
>>> Hi, All
>>>
>>> Thanks for the helpful comments!
>>> Nice to see this happens rarely in other environments.
>>>
>>> Actually I've changed the configuration not to run the task on the master node,
>>> but the same problem happened.
>>>
>>> So at first, upgrade the switch. Report again if the problem will be fixed.
>>>
>>> Thanks
>>> Kazuki
>>>
>>> On Thu, Apr 21, 2011 at 5:32 AM, Gary Helmling <gh...@gmail.com> wrote:
>>>>> I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15).
>>>>> The configuraiton is below.
>>>>>
>>>>> hdp0: zk + master + region + nn + dn + jt + tt
>>>>> hdp1: zk + master + region + snn + dn + tt
>>>>> hdp2: zk + region + dn + tt
>>>>> hdp3 to hdp15: region + dn + tt
>>>>>
>>>>>
>>>> I would also look at the memory configuration for your servers and the
>>>> amount of heap allocated to each process. Is it possible hdp0 is swapping
>>>> when running a MR job? Swapping will cause big headaches and is often a
>>>> culprit for zk session timeouts.
>>>>
>>>> Between the 7 processes it has plus any child tasks started, it's not hard
>>>> to picture overcommitting memory.
>>>>
>>>> Regardless of whether the core problem lies in network hardware or here, I
>>>> would remove the region server, data node, and task tracker processes from
>>>> hdp0 and hdp1 for smoother operation.
>>>>
>>>> --gh
>>>>
>>>
>>>
>>>
>>> --
>>> --------------------------------------------------
>>> Kazuki Ohta: http://kzk9.net/
>>> CTO at Preferred Infrastructure: http://preferred.jp/
>>>
>>
>>
>>
>> --
>> --------------------------------------------------
>> Kazuki Ohta: http://kzk9.net/
>> CTO at Preferred Infrastructure: http://preferred.jp/
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
--------------------------------------------------
Kazuki Ohta: http://kzk9.net/
CTO at Preferred Infrastructure: http://preferred.jp/
Re: massive zk expirations under heavy network load
Posted by Todd Lipcon <to...@cloudera.com>.
こんいちは大田さん ：） (still practicing Japanese over here)
Could you paste the output from "lspci" to the list? It would be
useful to know which particular hardware you had this problem with, so
we can watch out for it and know to upgrade e1000
Thanks
-Todd
On Wed, May 18, 2011 at 9:02 PM, Kazuki Ohta <ka...@gmail.com> wrote:
> Hi, all
>
> Finally got my cluster stable by upgrading network driver, instead of
> changing the
> switch.
>
> We used e1000e driver on CentOS 5.5, then upgrading to e1000e-1.3.10a.tar.gz,
> the most recent one. This dramatically reduces the # of dropped packets under
> heavy load.
>
>> http://bit.ly/lTJsV1
>
> Thanks for the help!
> Kazuki
>
> On Thu, Apr 21, 2011 at 11:24 AM, Kazuki Ohta <ka...@gmail.com> wrote:
>> Hi, All
>>
>> Thanks for the helpful comments!
>> Nice to see this happens rarely in other environments.
>>
>> Actually I've changed the configuration not to run the task on the master node,
>> but the same problem happened.
>>
>> So at first, upgrade the switch. Report again if the problem will be fixed.
>>
>> Thanks
>> Kazuki
>>
>> On Thu, Apr 21, 2011 at 5:32 AM, Gary Helmling <gh...@gmail.com> wrote:
>>>> I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15).
>>>> The configuraiton is below.
>>>>
>>>> hdp0: zk + master + region + nn + dn + jt + tt
>>>> hdp1: zk + master + region + snn + dn + tt
>>>> hdp2: zk + region + dn + tt
>>>> hdp3 to hdp15: region + dn + tt
>>>>
>>>>
>>> I would also look at the memory configuration for your servers and the
>>> amount of heap allocated to each process. Is it possible hdp0 is swapping
>>> when running a MR job? Swapping will cause big headaches and is often a
>>> culprit for zk session timeouts.
>>>
>>> Between the 7 processes it has plus any child tasks started, it's not hard
>>> to picture overcommitting memory.
>>>
>>> Regardless of whether the core problem lies in network hardware or here, I
>>> would remove the region server, data node, and task tracker processes from
>>> hdp0 and hdp1 for smoother operation.
>>>
>>> --gh
>>>
>>
>>
>>
>> --
>> --------------------------------------------------
>> Kazuki Ohta: http://kzk9.net/
>> CTO at Preferred Infrastructure: http://preferred.jp/
>>
>
>
>
> --
> --------------------------------------------------
> Kazuki Ohta: http://kzk9.net/
> CTO at Preferred Infrastructure: http://preferred.jp/
>
--
Todd Lipcon
Software Engineer, Cloudera