You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zakee <kz...@netzero.net> on 2015/03/02 21:53:03 UTC
Re: NetworkProcessorAvgIdlePercent
Thanks, I have added them for monitoring.
-Zakee
> On Feb 27, 2015, at 9:21 AM, Jun Rao <ju...@confluent.io> wrote:
>
> Zakee,
>
> It would be useful to get the following.
>
> kafka.network:name=RequestQueueSize,type=RequestChannel
> kafka.network:name=RequestQueueTimeMs,request=Fetch,type=RequestMetrics
> kafka.network:name=RequestQueueTimeMs,request=Produce,type=RequestMetrics
>
> Thanks,
>
> Jun
>
>
> On Thu, Feb 26, 2015 at 2:17 PM, Zakee <kzakee1@netzero.net <ma...@netzero.net>> wrote:
>
>> Hi Jun,
>>
>> There are too many KPIs generated by kafka that I can’t monitor them all,
>> I have monitoring a filtered list that I can understand so far.
>>
>> Which keys are you talking of? Currently I am not monitoring any of below
>> keys but will add only those that are useful.
>>
>> kafka.network:name=RequestQueueSize,type=RequestChannel
>>
>> kafka.network:name=RequestQueueTimeMs,request=ConsumerMetadata,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=ControlledShutdown,type=RequestMetrics
>> kafka.network:name=RequestQueueTimeMs,request=Fetch,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=FetchConsumer,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=FetchFollower,type=RequestMetrics
>> kafka.network:name=RequestQueueTimeMs,request=Heartbeat,type=RequestMetrics
>> kafka.network:name=RequestQueueTimeMs,request=JoinGroup,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=LeaderAndIsr,type=RequestMetrics
>> kafka.network:name=RequestQueueTimeMs,request=Metadata,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=OffsetCommit,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=OffsetFetch,type=RequestMetrics
>> kafka.network:name=RequestQueueTimeMs,request=Offsets,type=RequestMetrics
>> kafka.network:name=RequestQueueTimeMs,request=Produce,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=StopReplica,type=RequestMetrics
>>
>> kafka.network:name=RequestQueueTimeMs,request=UpdateMetadata,type=RequestMetrics
>>
>>
>> Thanks,
>> Zakee
>>
>>> On Feb 26, 2015, at 9:10 AM, Jun Rao <ju...@confluent.io> wrote:
>>>
>>> That may be enough. What's the RequestQueueSize and RequestQueueTimeMs?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Wed, Feb 25, 2015 at 10:24 PM, Zakee <kz...@netzero.net> wrote:
>>>
>>>> Well currently I have configured 14 thread for both io and network. Do
>> you
>>>> think we should consider more?
>>>>
>>>> Thanks
>>>> -Zakee
>>>>
>>>> On Wed, Feb 25, 2015 at 6:22 PM, Jun Rao <ju...@confluent.io> wrote:
>>>>
>>>>> Then you may want to consider increasing num.io.threads
>>>>> and num.network.threads.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Tue, Feb 24, 2015 at 7:48 PM, Zakee <kz...@netzero.net> wrote:
>>>>>
>>>>>> Similar pattern for that too. Mostly hovering below.
>>>>>>
>>>>>> -Zakee
>>>>>>
>>>>>> On Tue, Feb 24, 2015 at 2:43 PM, Jun Rao <ju...@confluent.io> wrote:
>>>>>>
>>>>>>> What about RequestHandlerAvgIdlePercent?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 8:47 PM, Zakee <kz...@netzero.net> wrote:
>>>>>>>
>>>>>>>> Hi Jun,
>>>>>>>>
>>>>>>>> With ~100G of data being pushed per hour across 35 topics
>>>>>>>> (replication-factor 3), the NetworkProcessorAvgIdlePercent is
>>>> mostly
>>>>>>>> showing below 0.5 sometimes when the producers send on a high rate.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> -Zakee
>>>>>>>>
>>>>>>>> On Sun, Feb 22, 2015 at 10:29 PM, Jun Rao <ju...@confluent.io>
>>>> wrote:
>>>>>>>>
>>>>>>>>> What kind of load do you have on the brokers? On an idle cluster
>>>>>> (just
>>>>>>>>> fetch requests from the follower replicas), I
>>>>>>>>> saw NetworkProcessorAvgIdlePercent at about 97%.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>> On Thu, Feb 19, 2015 at 5:19 PM, Zakee <kz...@netzero.net>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Jun,
>>>>>>>>>>
>>>>>>>>>> I am already using the latest release 0.8.2.1.
>>>>>>>>>>
>>>>>>>>>> -Zakee
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 19, 2015 at 2:46 PM, Jun Rao <ju...@confluent.io>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Could you try the 0.8.2.1 release being voted on now? It
>>>> fixes
>>>>> a
>>>>>>> CPU
>>>>>>>>>> issue
>>>>>>>>>>> and should reduce the CPU load in network thread.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Jun
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 19, 2015 at 11:54 AM, Zakee <kzakee1@netzero.net
>>>>>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Kafka documentation recommends <0.3 for above metric. I
>>>>> assume
>>>>>>>>>> processor
>>>>>>>>>>> is
>>>>>>>>>>>> busier if this goes below 0.3 and obviously it being < 0.3
>>>>> for
>>>>>>> long
>>>>>>>>>> does
>>>>>>>>>>>> not seem to be a good sign.
>>>>>>>>>>>>
>>>>>>>>>>>> What should be our criteria to raise an alert, I though it
>>>>>> should
>>>>>>>> be
>>>>>>>>>>> when
>>>>>>>>>>>> its value goes below 0.3. However, the value seems to be
>>>>> below
>>>>>>> 0.3
>>>>>>>> a
>>>>>>>>>> lot
>>>>>>>>>>> of
>>>>>>>>>>>> the times, almost always if we take samples every five
>>>> mins.
>>>>>> What
>>>>>>>>>> should
>>>>>>>>>>> be
>>>>>>>>>>>> the threshold to raise an alarm ?
>>>>>>>>>>>>
>>>>>>>>>>>> What would be the impact of having this below 0.3 or even
>>>>> zero
>>>>>>> like
>>>>>>>>>> most
>>>>>>>>>>> of
>>>>>>>>>>>> the times?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -Zakee
>>>>>>>>>>>>
>>>> ____________________________________________________________
>>>>>>>>>>>> How Old Men Tighten Skin
>>>>>>>>>>>> 63 Year Old Man Shares DIY Skin Tightening Method You Can
>>>> Do
>>>>>> From
>>>>>>>>> Home
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> http://thirdpartyoffers.netzero.net/TGL3231/54e63f5bda4c23f5b6560st02vuc
>>>>>>>>>>> ____________________________________________________________
>>>>>>>>>>> 8% Annuity Return Secret
>>>>>>>>>>> Earn Guaranteed Income for Life! Compare Rates Today.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> http://thirdpartyoffers.netzero.net/TGL3255/54e6782bcbe78782b37bdmp15duc
>>>>>>>>>>
>>>>>>>>> ____________________________________________________________
>>>>>>>>> High School Yearbooks
>>>>>>>>> View Class Yearbooks Online Free. Reminisce & Buy a Reprint
>>>> Today!
>>>>>>>>>
>>>>>>>
>>>>>
>> http://thirdpartyoffers.netzero.net/TGL3255/54eac962abad049627e56mp03duc
>>>>>>>>
>>>>>>> ____________________________________________________________
>>>>>>> Extended Stay America
>>>>>>> Official Site. Free WIFI, Kitchens. Our best rates here, guaranteed.
>>>>>>>
>>>>>
>> http://thirdpartyoffers.netzero.net/TGL3255/54ed04fb9538c4fb6188mp15duc
>>>>>>
>>>>> ____________________________________________________________
>>>>> Extended Stay America
>>>>> Official Site. Free WIFI, Kitchens. Our best rates here, guaranteed.
>>>>>
>> http://thirdpartyoffers.netzero.net/TGL3255/54ee8dd7e1cd7dd777e8mp03duc
>>>>
>>> ____________________________________________________________
>>> Protect what matters
>>> Floods can happen anywhere. Learn your risk and find an agent today.
>>> http://thirdpartyoffers.netzero.net/TGL3255/54ef551637a8b5514471fmp01duc
>>
>>
> ____________________________________________________________
> What's your flood risk?
> Find flood maps, interactive tools, FAQs, and agents in your area.
> http://thirdpartyoffers.netzero.net/TGL3255/54f0ad572e6932d5774b1mp08duc <http://thirdpartyoffers.netzero.net/TGL3255/54f0ad572e6932d5774b1mp08duc>