You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Krish Donald <go...@gmail.com> on 2015/03/02 05:45:03 UTC
How to find bottlenecks of the cluster ?
Hi,
I wanted to understand, how should we find out the bottleneck of the
cluster?
Thanks
Krish
Re: How to find bottlenecks of the cluster ?
Posted by 杨浩 <ya...@gmail.com>.
I think benchmark will do some help, since it can help to find out the
executing speed of I/O rated job and CPU rated job
2015-03-02 19:01 GMT+08:00 Adrien Mogenet <ad...@contentsquare.com>
:
> This is a non-sense ; you have to tell us under which conditions you want
> to find a bottleneck.
>
> Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
> / user / sys / idle), disk usage (await, ios in progress...) and memory
> (numa allocations, buffers, cache, dirty pages...)
>
> On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
>
>> Basically we have 4 points to consider, CPU , Memory, IO and Network
>>
>> So how to see which one is causing the bottleneck ?
>> What parameters we should consider etc ?
>>
>> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
>> wrote:
>>
>>> This is a vast topic.Can you tell what components are there in your
>>> data pipe line and how data flows in to system and the way its
>>> processed.There are several inbuilt tests like testDFSIO and terasort that
>>> you can run.
>>>
>>> -Nishan
>>>
>>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I wanted to understand, how should we find out the bottleneck of the
>>>> cluster?
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>
>>>
>>
>
>
> --
>
> *Adrien Mogenet*
> Head of Backend/Infrastructure
> adrien.mogenet@contentsquare.com
> (+33)6.59.16.64.22
> http://www.contentsquare.com
> 4, avenue Franklin D. Roosevelt - 75008 Paris
>
Re: How to find bottlenecks of the cluster ?
Posted by 杨浩 <ya...@gmail.com>.
I think benchmark will do some help, since it can help to find out the
executing speed of I/O rated job and CPU rated job
2015-03-02 19:01 GMT+08:00 Adrien Mogenet <ad...@contentsquare.com>
:
> This is a non-sense ; you have to tell us under which conditions you want
> to find a bottleneck.
>
> Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
> / user / sys / idle), disk usage (await, ios in progress...) and memory
> (numa allocations, buffers, cache, dirty pages...)
>
> On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
>
>> Basically we have 4 points to consider, CPU , Memory, IO and Network
>>
>> So how to see which one is causing the bottleneck ?
>> What parameters we should consider etc ?
>>
>> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
>> wrote:
>>
>>> This is a vast topic.Can you tell what components are there in your
>>> data pipe line and how data flows in to system and the way its
>>> processed.There are several inbuilt tests like testDFSIO and terasort that
>>> you can run.
>>>
>>> -Nishan
>>>
>>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I wanted to understand, how should we find out the bottleneck of the
>>>> cluster?
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>
>>>
>>
>
>
> --
>
> *Adrien Mogenet*
> Head of Backend/Infrastructure
> adrien.mogenet@contentsquare.com
> (+33)6.59.16.64.22
> http://www.contentsquare.com
> 4, avenue Franklin D. Roosevelt - 75008 Paris
>
Re: How to find bottlenecks of the cluster ?
Posted by 杨浩 <ya...@gmail.com>.
I think benchmark will do some help, since it can help to find out the
executing speed of I/O rated job and CPU rated job
2015-03-02 19:01 GMT+08:00 Adrien Mogenet <ad...@contentsquare.com>
:
> This is a non-sense ; you have to tell us under which conditions you want
> to find a bottleneck.
>
> Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
> / user / sys / idle), disk usage (await, ios in progress...) and memory
> (numa allocations, buffers, cache, dirty pages...)
>
> On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
>
>> Basically we have 4 points to consider, CPU , Memory, IO and Network
>>
>> So how to see which one is causing the bottleneck ?
>> What parameters we should consider etc ?
>>
>> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
>> wrote:
>>
>>> This is a vast topic.Can you tell what components are there in your
>>> data pipe line and how data flows in to system and the way its
>>> processed.There are several inbuilt tests like testDFSIO and terasort that
>>> you can run.
>>>
>>> -Nishan
>>>
>>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I wanted to understand, how should we find out the bottleneck of the
>>>> cluster?
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>
>>>
>>
>
>
> --
>
> *Adrien Mogenet*
> Head of Backend/Infrastructure
> adrien.mogenet@contentsquare.com
> (+33)6.59.16.64.22
> http://www.contentsquare.com
> 4, avenue Franklin D. Roosevelt - 75008 Paris
>
Re: How to find bottlenecks of the cluster ?
Posted by 杨浩 <ya...@gmail.com>.
I think benchmark will do some help, since it can help to find out the
executing speed of I/O rated job and CPU rated job
2015-03-02 19:01 GMT+08:00 Adrien Mogenet <ad...@contentsquare.com>
:
> This is a non-sense ; you have to tell us under which conditions you want
> to find a bottleneck.
>
> Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
> / user / sys / idle), disk usage (await, ios in progress...) and memory
> (numa allocations, buffers, cache, dirty pages...)
>
> On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
>
>> Basically we have 4 points to consider, CPU , Memory, IO and Network
>>
>> So how to see which one is causing the bottleneck ?
>> What parameters we should consider etc ?
>>
>> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
>> wrote:
>>
>>> This is a vast topic.Can you tell what components are there in your
>>> data pipe line and how data flows in to system and the way its
>>> processed.There are several inbuilt tests like testDFSIO and terasort that
>>> you can run.
>>>
>>> -Nishan
>>>
>>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I wanted to understand, how should we find out the bottleneck of the
>>>> cluster?
>>>>
>>>> Thanks
>>>> Krish
>>>>
>>>
>>>
>>
>
>
> --
>
> *Adrien Mogenet*
> Head of Backend/Infrastructure
> adrien.mogenet@contentsquare.com
> (+33)6.59.16.64.22
> http://www.contentsquare.com
> 4, avenue Franklin D. Roosevelt - 75008 Paris
>
Re: How to find bottlenecks of the cluster ?
Posted by Adrien Mogenet <ad...@contentsquare.com>.
This is a non-sense ; you have to tell us under which conditions you want
to find a bottleneck.
Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
/ user / sys / idle), disk usage (await, ios in progress...) and memory
(numa allocations, buffers, cache, dirty pages...)
On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
> Basically we have 4 points to consider, CPU , Memory, IO and Network
>
> So how to see which one is causing the bottleneck ?
> What parameters we should consider etc ?
>
> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
> wrote:
>
>> This is a vast topic.Can you tell what components are there in your
>> data pipe line and how data flows in to system and the way its
>> processed.There are several inbuilt tests like testDFSIO and terasort that
>> you can run.
>>
>> -Nishan
>>
>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I wanted to understand, how should we find out the bottleneck of the
>>> cluster?
>>>
>>> Thanks
>>> Krish
>>>
>>
>>
>
--
*Adrien Mogenet*
Head of Backend/Infrastructure
adrien.mogenet@contentsquare.com
(+33)6.59.16.64.22
http://www.contentsquare.com
4, avenue Franklin D. Roosevelt - 75008 Paris
Re: How to find bottlenecks of the cluster ?
Posted by Adrien Mogenet <ad...@contentsquare.com>.
This is a non-sense ; you have to tell us under which conditions you want
to find a bottleneck.
Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
/ user / sys / idle), disk usage (await, ios in progress...) and memory
(numa allocations, buffers, cache, dirty pages...)
On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
> Basically we have 4 points to consider, CPU , Memory, IO and Network
>
> So how to see which one is causing the bottleneck ?
> What parameters we should consider etc ?
>
> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
> wrote:
>
>> This is a vast topic.Can you tell what components are there in your
>> data pipe line and how data flows in to system and the way its
>> processed.There are several inbuilt tests like testDFSIO and terasort that
>> you can run.
>>
>> -Nishan
>>
>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I wanted to understand, how should we find out the bottleneck of the
>>> cluster?
>>>
>>> Thanks
>>> Krish
>>>
>>
>>
>
--
*Adrien Mogenet*
Head of Backend/Infrastructure
adrien.mogenet@contentsquare.com
(+33)6.59.16.64.22
http://www.contentsquare.com
4, avenue Franklin D. Roosevelt - 75008 Paris
Re: How to find bottlenecks of the cluster ?
Posted by Adrien Mogenet <ad...@contentsquare.com>.
This is a non-sense ; you have to tell us under which conditions you want
to find a bottleneck.
Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
/ user / sys / idle), disk usage (await, ios in progress...) and memory
(numa allocations, buffers, cache, dirty pages...)
On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
> Basically we have 4 points to consider, CPU , Memory, IO and Network
>
> So how to see which one is causing the bottleneck ?
> What parameters we should consider etc ?
>
> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
> wrote:
>
>> This is a vast topic.Can you tell what components are there in your
>> data pipe line and how data flows in to system and the way its
>> processed.There are several inbuilt tests like testDFSIO and terasort that
>> you can run.
>>
>> -Nishan
>>
>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I wanted to understand, how should we find out the bottleneck of the
>>> cluster?
>>>
>>> Thanks
>>> Krish
>>>
>>
>>
>
--
*Adrien Mogenet*
Head of Backend/Infrastructure
adrien.mogenet@contentsquare.com
(+33)6.59.16.64.22
http://www.contentsquare.com
4, avenue Franklin D. Roosevelt - 75008 Paris
Re: How to find bottlenecks of the cluster ?
Posted by Adrien Mogenet <ad...@contentsquare.com>.
This is a non-sense ; you have to tell us under which conditions you want
to find a bottleneck.
Regardless the workload, we mostly use OpenTSDB to check cpu times (iowait
/ user / sys / idle), disk usage (await, ios in progress...) and memory
(numa allocations, buffers, cache, dirty pages...)
On 2 March 2015 at 08:20, Krish Donald <go...@gmail.com> wrote:
> Basically we have 4 points to consider, CPU , Memory, IO and Network
>
> So how to see which one is causing the bottleneck ?
> What parameters we should consider etc ?
>
> On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com>
> wrote:
>
>> This is a vast topic.Can you tell what components are there in your
>> data pipe line and how data flows in to system and the way its
>> processed.There are several inbuilt tests like testDFSIO and terasort that
>> you can run.
>>
>> -Nishan
>>
>> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I wanted to understand, how should we find out the bottleneck of the
>>> cluster?
>>>
>>> Thanks
>>> Krish
>>>
>>
>>
>
--
*Adrien Mogenet*
Head of Backend/Infrastructure
adrien.mogenet@contentsquare.com
(+33)6.59.16.64.22
http://www.contentsquare.com
4, avenue Franklin D. Roosevelt - 75008 Paris
Re: How to find bottlenecks of the cluster ?
Posted by Krish Donald <go...@gmail.com>.
Basically we have 4 points to consider, CPU , Memory, IO and Network
So how to see which one is causing the bottleneck ?
What parameters we should consider etc ?
On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com> wrote:
> This is a vast topic.Can you tell what components are there in your data
> pipe line and how data flows in to system and the way its processed.There
> are several inbuilt tests like testDFSIO and terasort that you can run.
>
> -Nishan
>
> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
>
>> Hi,
>>
>> I wanted to understand, how should we find out the bottleneck of the
>> cluster?
>>
>> Thanks
>> Krish
>>
>
>
Re: How to find bottlenecks of the cluster ?
Posted by Krish Donald <go...@gmail.com>.
Basically we have 4 points to consider, CPU , Memory, IO and Network
So how to see which one is causing the bottleneck ?
What parameters we should consider etc ?
On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com> wrote:
> This is a vast topic.Can you tell what components are there in your data
> pipe line and how data flows in to system and the way its processed.There
> are several inbuilt tests like testDFSIO and terasort that you can run.
>
> -Nishan
>
> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
>
>> Hi,
>>
>> I wanted to understand, how should we find out the bottleneck of the
>> cluster?
>>
>> Thanks
>> Krish
>>
>
>
Re: How to find bottlenecks of the cluster ?
Posted by Krish Donald <go...@gmail.com>.
Basically we have 4 points to consider, CPU , Memory, IO and Network
So how to see which one is causing the bottleneck ?
What parameters we should consider etc ?
On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com> wrote:
> This is a vast topic.Can you tell what components are there in your data
> pipe line and how data flows in to system and the way its processed.There
> are several inbuilt tests like testDFSIO and terasort that you can run.
>
> -Nishan
>
> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
>
>> Hi,
>>
>> I wanted to understand, how should we find out the bottleneck of the
>> cluster?
>>
>> Thanks
>> Krish
>>
>
>
Re: How to find bottlenecks of the cluster ?
Posted by Krish Donald <go...@gmail.com>.
Basically we have 4 points to consider, CPU , Memory, IO and Network
So how to see which one is causing the bottleneck ?
What parameters we should consider etc ?
On Sun, Mar 1, 2015 at 10:57 PM, Nishanth S <ni...@gmail.com> wrote:
> This is a vast topic.Can you tell what components are there in your data
> pipe line and how data flows in to system and the way its processed.There
> are several inbuilt tests like testDFSIO and terasort that you can run.
>
> -Nishan
>
> On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
>
>> Hi,
>>
>> I wanted to understand, how should we find out the bottleneck of the
>> cluster?
>>
>> Thanks
>> Krish
>>
>
>
Re: How to find bottlenecks of the cluster ?
Posted by Nishanth S <ni...@gmail.com>.
This is a vast topic.Can you tell what components are there in your data
pipe line and how data flows in to system and the way its processed.There
are several inbuilt tests like testDFSIO and terasort that you can run.
-Nishan
On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
> Hi,
>
> I wanted to understand, how should we find out the bottleneck of the
> cluster?
>
> Thanks
> Krish
>
Re: How to find bottlenecks of the cluster ?
Posted by Nishanth S <ni...@gmail.com>.
This is a vast topic.Can you tell what components are there in your data
pipe line and how data flows in to system and the way its processed.There
are several inbuilt tests like testDFSIO and terasort that you can run.
-Nishan
On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
> Hi,
>
> I wanted to understand, how should we find out the bottleneck of the
> cluster?
>
> Thanks
> Krish
>
Re: How to find bottlenecks of the cluster ?
Posted by Nishanth S <ni...@gmail.com>.
This is a vast topic.Can you tell what components are there in your data
pipe line and how data flows in to system and the way its processed.There
are several inbuilt tests like testDFSIO and terasort that you can run.
-Nishan
On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
> Hi,
>
> I wanted to understand, how should we find out the bottleneck of the
> cluster?
>
> Thanks
> Krish
>
Re: How to find bottlenecks of the cluster ?
Posted by Nishanth S <ni...@gmail.com>.
This is a vast topic.Can you tell what components are there in your data
pipe line and how data flows in to system and the way its processed.There
are several inbuilt tests like testDFSIO and terasort that you can run.
-Nishan
On Sun, Mar 1, 2015 at 9:45 PM, Krish Donald <go...@gmail.com> wrote:
> Hi,
>
> I wanted to understand, how should we find out the bottleneck of the
> cluster?
>
> Thanks
> Krish
>