You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jinal Shah <ji...@gmail.com> on 2014/10/01 16:58:21 UTC
WARN messages clarifications
Hi everyone
We are seeing these WARN log messages from this class
com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
It normally happens when the query takes a long time to execute and
displays the duration as well like this
Duration Threshold exceeded: [durationThreshold=1000]
Actuals: [actualBatchSize=1: actualPayloadSize=0: actualDuration=3929]
The scenarios I found for the delay are
- Network issues taking time for delays
- Trying to establish a connection for the first time
- Some component failures occurred like Zookeeper node when down or
Master failed over.
So can someone give say what other possible scenarios for delays in
executing a query.
Thanks
Jinal
Re: WARN messages clarifications
Posted by Ted Yu <yu...@gmail.com>.
Not so much for minor compactions.
For major compactions, yes.
See this thread
http://search-hadoop.com/m/Bg1JmK8OGb/compaction+hbase+query&subj=difference+between+major+and+minor+compactions+
On Wed, Oct 1, 2014 at 5:05 PM, Jinal Shah <ji...@gmail.com> wrote:
> Just a follow-up on this. Does compaction add up to query execution time in
> hbase?
>
> On Wed, Oct 1, 2014 at 1:34 PM, Jinal Shah <ji...@gmail.com>
> wrote:
>
> > Thanks Ted, that helps.
> >
> > On Wed, Oct 1, 2014 at 12:51 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Suppose your OutlierHTable was communicating with a region server which
> >> happened to go through GC pause.
> >> The query would exhibit longer duration compared to one where no region
> >> server goes through GC (assuming other factors being equal: network
> >> traffic, size of data queried, etc).
> >>
> >> bq. Is GC pause set on a timer?
> >>
> >> I don't think so. It depends on the load region server handles.
> >>
> >> Cheers
> >>
> >> On Wed, Oct 1, 2014 at 10:37 AM, Jinal Shah <ji...@gmail.com>
> >> wrote:
> >>
> >> > Is GC pause set on a timer? How much does that impact the query
> request?
> >> >
> >> > On Wed, Oct 1, 2014 at 10:48 AM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> > > One other possibility would be the region server going through GC
> >> pause.
> >> > >
> >> > > Cheers
> >> > >
> >> > > On Wed, Oct 1, 2014 at 8:04 AM, Jinal Shah <jinalshah2007@gmail.com
> >
> >> > > wrote:
> >> > >
> >> > > > We are using 0.94.15 version.
> >> > > >
> >> > > > On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > I searched 0.94 and 0.98 code base but didn't find the message
> >> below.
> >> > > > >
> >> > > > > Which version of HBase are you using ?
> >> > > > >
> >> > > > > Cheers
> >> > > > >
> >> > > > > On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <
> >> jinalshah2007@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi everyone
> >> > > > > >
> >> > > > > > We are seeing these WARN log messages from this class
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> >> > > > > > It normally happens when the query takes a long time to
> execute
> >> and
> >> > > > > > displays the duration as well like this
> >> > > > > >
> >> > > > > > Duration Threshold exceeded: [durationThreshold=1000]
> >> > > > > > Actuals: [actualBatchSize=1: actualPayloadSize=0:
> >> > > actualDuration=3929]
> >> > > > > >
> >> > > > > > The scenarios I found for the delay are
> >> > > > > >
> >> > > > > >
> >> > > > > > - Network issues taking time for delays
> >> > > > > > - Trying to establish a connection for the first time
> >> > > > > > - Some component failures occurred like Zookeeper node when
> >> down
> >> > > or
> >> > > > > > Master failed over.
> >> > > > > >
> >> > > > > >
> >> > > > > > So can someone give say what other possible scenarios for
> >> delays in
> >> > > > > > executing a query.
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > > Jinal
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: WARN messages clarifications
Posted by Jinal Shah <ji...@gmail.com>.
Just a follow-up on this. Does compaction add up to query execution time in
hbase?
On Wed, Oct 1, 2014 at 1:34 PM, Jinal Shah <ji...@gmail.com> wrote:
> Thanks Ted, that helps.
>
> On Wed, Oct 1, 2014 at 12:51 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Suppose your OutlierHTable was communicating with a region server which
>> happened to go through GC pause.
>> The query would exhibit longer duration compared to one where no region
>> server goes through GC (assuming other factors being equal: network
>> traffic, size of data queried, etc).
>>
>> bq. Is GC pause set on a timer?
>>
>> I don't think so. It depends on the load region server handles.
>>
>> Cheers
>>
>> On Wed, Oct 1, 2014 at 10:37 AM, Jinal Shah <ji...@gmail.com>
>> wrote:
>>
>> > Is GC pause set on a timer? How much does that impact the query request?
>> >
>> > On Wed, Oct 1, 2014 at 10:48 AM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > One other possibility would be the region server going through GC
>> pause.
>> > >
>> > > Cheers
>> > >
>> > > On Wed, Oct 1, 2014 at 8:04 AM, Jinal Shah <ji...@gmail.com>
>> > > wrote:
>> > >
>> > > > We are using 0.94.15 version.
>> > > >
>> > > > On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>> > > >
>> > > > > I searched 0.94 and 0.98 code base but didn't find the message
>> below.
>> > > > >
>> > > > > Which version of HBase are you using ?
>> > > > >
>> > > > > Cheers
>> > > > >
>> > > > > On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <
>> jinalshah2007@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi everyone
>> > > > > >
>> > > > > > We are seeing these WARN log messages from this class
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
>> > > > > > It normally happens when the query takes a long time to execute
>> and
>> > > > > > displays the duration as well like this
>> > > > > >
>> > > > > > Duration Threshold exceeded: [durationThreshold=1000]
>> > > > > > Actuals: [actualBatchSize=1: actualPayloadSize=0:
>> > > actualDuration=3929]
>> > > > > >
>> > > > > > The scenarios I found for the delay are
>> > > > > >
>> > > > > >
>> > > > > > - Network issues taking time for delays
>> > > > > > - Trying to establish a connection for the first time
>> > > > > > - Some component failures occurred like Zookeeper node when
>> down
>> > > or
>> > > > > > Master failed over.
>> > > > > >
>> > > > > >
>> > > > > > So can someone give say what other possible scenarios for
>> delays in
>> > > > > > executing a query.
>> > > > > >
>> > > > > > Thanks
>> > > > > > Jinal
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: WARN messages clarifications
Posted by Jinal Shah <ji...@gmail.com>.
Thanks Ted, that helps.
On Wed, Oct 1, 2014 at 12:51 PM, Ted Yu <yu...@gmail.com> wrote:
> Suppose your OutlierHTable was communicating with a region server which
> happened to go through GC pause.
> The query would exhibit longer duration compared to one where no region
> server goes through GC (assuming other factors being equal: network
> traffic, size of data queried, etc).
>
> bq. Is GC pause set on a timer?
>
> I don't think so. It depends on the load region server handles.
>
> Cheers
>
> On Wed, Oct 1, 2014 at 10:37 AM, Jinal Shah <ji...@gmail.com>
> wrote:
>
> > Is GC pause set on a timer? How much does that impact the query request?
> >
> > On Wed, Oct 1, 2014 at 10:48 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > One other possibility would be the region server going through GC
> pause.
> > >
> > > Cheers
> > >
> > > On Wed, Oct 1, 2014 at 8:04 AM, Jinal Shah <ji...@gmail.com>
> > > wrote:
> > >
> > > > We are using 0.94.15 version.
> > > >
> > > > On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > I searched 0.94 and 0.98 code base but didn't find the message
> below.
> > > > >
> > > > > Which version of HBase are you using ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <
> jinalshah2007@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi everyone
> > > > > >
> > > > > > We are seeing these WARN log messages from this class
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> > > > > > It normally happens when the query takes a long time to execute
> and
> > > > > > displays the duration as well like this
> > > > > >
> > > > > > Duration Threshold exceeded: [durationThreshold=1000]
> > > > > > Actuals: [actualBatchSize=1: actualPayloadSize=0:
> > > actualDuration=3929]
> > > > > >
> > > > > > The scenarios I found for the delay are
> > > > > >
> > > > > >
> > > > > > - Network issues taking time for delays
> > > > > > - Trying to establish a connection for the first time
> > > > > > - Some component failures occurred like Zookeeper node when
> down
> > > or
> > > > > > Master failed over.
> > > > > >
> > > > > >
> > > > > > So can someone give say what other possible scenarios for delays
> in
> > > > > > executing a query.
> > > > > >
> > > > > > Thanks
> > > > > > Jinal
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: WARN messages clarifications
Posted by Ted Yu <yu...@gmail.com>.
Suppose your OutlierHTable was communicating with a region server which
happened to go through GC pause.
The query would exhibit longer duration compared to one where no region
server goes through GC (assuming other factors being equal: network
traffic, size of data queried, etc).
bq. Is GC pause set on a timer?
I don't think so. It depends on the load region server handles.
Cheers
On Wed, Oct 1, 2014 at 10:37 AM, Jinal Shah <ji...@gmail.com> wrote:
> Is GC pause set on a timer? How much does that impact the query request?
>
> On Wed, Oct 1, 2014 at 10:48 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > One other possibility would be the region server going through GC pause.
> >
> > Cheers
> >
> > On Wed, Oct 1, 2014 at 8:04 AM, Jinal Shah <ji...@gmail.com>
> > wrote:
> >
> > > We are using 0.94.15 version.
> > >
> > > On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > I searched 0.94 and 0.98 code base but didn't find the message below.
> > > >
> > > > Which version of HBase are you using ?
> > > >
> > > > Cheers
> > > >
> > > > On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <ji...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi everyone
> > > > >
> > > > > We are seeing these WARN log messages from this class
> > > > >
> > > >
> > >
> >
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> > > > > It normally happens when the query takes a long time to execute and
> > > > > displays the duration as well like this
> > > > >
> > > > > Duration Threshold exceeded: [durationThreshold=1000]
> > > > > Actuals: [actualBatchSize=1: actualPayloadSize=0:
> > actualDuration=3929]
> > > > >
> > > > > The scenarios I found for the delay are
> > > > >
> > > > >
> > > > > - Network issues taking time for delays
> > > > > - Trying to establish a connection for the first time
> > > > > - Some component failures occurred like Zookeeper node when down
> > or
> > > > > Master failed over.
> > > > >
> > > > >
> > > > > So can someone give say what other possible scenarios for delays in
> > > > > executing a query.
> > > > >
> > > > > Thanks
> > > > > Jinal
> > > > >
> > > >
> > >
> >
>
Re: WARN messages clarifications
Posted by Jinal Shah <ji...@gmail.com>.
Is GC pause set on a timer? How much does that impact the query request?
On Wed, Oct 1, 2014 at 10:48 AM, Ted Yu <yu...@gmail.com> wrote:
> One other possibility would be the region server going through GC pause.
>
> Cheers
>
> On Wed, Oct 1, 2014 at 8:04 AM, Jinal Shah <ji...@gmail.com>
> wrote:
>
> > We are using 0.94.15 version.
> >
> > On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I searched 0.94 and 0.98 code base but didn't find the message below.
> > >
> > > Which version of HBase are you using ?
> > >
> > > Cheers
> > >
> > > On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <ji...@gmail.com>
> > > wrote:
> > >
> > > > Hi everyone
> > > >
> > > > We are seeing these WARN log messages from this class
> > > >
> > >
> >
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> > > > It normally happens when the query takes a long time to execute and
> > > > displays the duration as well like this
> > > >
> > > > Duration Threshold exceeded: [durationThreshold=1000]
> > > > Actuals: [actualBatchSize=1: actualPayloadSize=0:
> actualDuration=3929]
> > > >
> > > > The scenarios I found for the delay are
> > > >
> > > >
> > > > - Network issues taking time for delays
> > > > - Trying to establish a connection for the first time
> > > > - Some component failures occurred like Zookeeper node when down
> or
> > > > Master failed over.
> > > >
> > > >
> > > > So can someone give say what other possible scenarios for delays in
> > > > executing a query.
> > > >
> > > > Thanks
> > > > Jinal
> > > >
> > >
> >
>
Re: WARN messages clarifications
Posted by Ted Yu <yu...@gmail.com>.
One other possibility would be the region server going through GC pause.
Cheers
On Wed, Oct 1, 2014 at 8:04 AM, Jinal Shah <ji...@gmail.com> wrote:
> We are using 0.94.15 version.
>
> On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I searched 0.94 and 0.98 code base but didn't find the message below.
> >
> > Which version of HBase are you using ?
> >
> > Cheers
> >
> > On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <ji...@gmail.com>
> > wrote:
> >
> > > Hi everyone
> > >
> > > We are seeing these WARN log messages from this class
> > >
> >
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> > > It normally happens when the query takes a long time to execute and
> > > displays the duration as well like this
> > >
> > > Duration Threshold exceeded: [durationThreshold=1000]
> > > Actuals: [actualBatchSize=1: actualPayloadSize=0: actualDuration=3929]
> > >
> > > The scenarios I found for the delay are
> > >
> > >
> > > - Network issues taking time for delays
> > > - Trying to establish a connection for the first time
> > > - Some component failures occurred like Zookeeper node when down or
> > > Master failed over.
> > >
> > >
> > > So can someone give say what other possible scenarios for delays in
> > > executing a query.
> > >
> > > Thanks
> > > Jinal
> > >
> >
>
Re: WARN messages clarifications
Posted by Jinal Shah <ji...@gmail.com>.
We are using 0.94.15 version.
On Wed, Oct 1, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
> I searched 0.94 and 0.98 code base but didn't find the message below.
>
> Which version of HBase are you using ?
>
> Cheers
>
> On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <ji...@gmail.com>
> wrote:
>
> > Hi everyone
> >
> > We are seeing these WARN log messages from this class
> >
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> > It normally happens when the query takes a long time to execute and
> > displays the duration as well like this
> >
> > Duration Threshold exceeded: [durationThreshold=1000]
> > Actuals: [actualBatchSize=1: actualPayloadSize=0: actualDuration=3929]
> >
> > The scenarios I found for the delay are
> >
> >
> > - Network issues taking time for delays
> > - Trying to establish a connection for the first time
> > - Some component failures occurred like Zookeeper node when down or
> > Master failed over.
> >
> >
> > So can someone give say what other possible scenarios for delays in
> > executing a query.
> >
> > Thanks
> > Jinal
> >
>
Re: WARN messages clarifications
Posted by Ted Yu <yu...@gmail.com>.
I searched 0.94 and 0.98 code base but didn't find the message below.
Which version of HBase are you using ?
Cheers
On Wed, Oct 1, 2014 at 7:58 AM, Jinal Shah <ji...@gmail.com> wrote:
> Hi everyone
>
> We are seeing these WARN log messages from this class
> com.cerner.hbase.extensions.OutlierHTableInterceptorFactory$OutlierHTable.
> It normally happens when the query takes a long time to execute and
> displays the duration as well like this
>
> Duration Threshold exceeded: [durationThreshold=1000]
> Actuals: [actualBatchSize=1: actualPayloadSize=0: actualDuration=3929]
>
> The scenarios I found for the delay are
>
>
> - Network issues taking time for delays
> - Trying to establish a connection for the first time
> - Some component failures occurred like Zookeeper node when down or
> Master failed over.
>
>
> So can someone give say what other possible scenarios for delays in
> executing a query.
>
> Thanks
> Jinal
>