You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by jeff saremi <je...@hotmail.com> on 2017/02/28 20:37:26 UTC
Need guidance on getting detailed elapsed times in every stage of
processing a request
I think we need to get detailed information from HBase RegionServer logs on how a request (read or write) is processed. Specifically speaking, i need to know of say 100 ms time spent in processing a write, how much of it was spent waiting for the HDFS?
What is the most efficient way of enabling this in log4j properties? Are there better mechanisms to get this information?
If I can get this in the log, then I can process the logs offline or in neartime and mount some dashboards on the top.
thanks
Jeff
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
thank you very much. this was much needed
________________________________
From: Yu Li <ca...@gmail.com>
Sent: Saturday, June 3, 2017 1:32:07 AM
To: dev@hbase.apache.org
Cc: jeff saremi
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
HBASE-15160 is done with great efforts from @Enis. You could check the latest patch there and do a customized backport if needed. @jeff
Best Regards,
Yu
On 16 March 2017 at 02:38, Enis Söztutar <en...@gmail.com>> wrote:
If I remember from the jira, it got stuck on perf testing the different
approaches of incrementing the histograms inline versus using a
multi-producer single consumer queue (blockingQueue, or Disruptor, etc),
and emitting the metrics inline and updating the histograms from a
background thread. Which approach is better should be tested.
Let's revive the jira.
Enis
On Tue, Mar 14, 2017 at 6:50 PM, Yu Li <ca...@gmail.com>> wrote:
> Let me check with @Enis in JIRA and get back to you (maybe days, due to
> schedule) later.
>
> Best Regards,
> Yu
>
> On 15 March 2017 at 05:57, jeff saremi <je...@hotmail.com>> wrote:
>
> > What's involved in getting this change merged into the main branch? These
> > 2 counters (fsReadLatency, fsWriteLatency) are super important to us
> > understanding what goes on behind every request. These are the minimum we
> > need to have especially in the absence of HTrace.
> >
> > I just checked the latest code and it does not seem to have these.
> >
> > thanks
> >
> > Jeff
> >
> >
> >
> > ------------------------------
> > *From:* jeff saremi <je...@hotmail.com>>
> > *Sent:* Sunday, March 5, 2017 3:41:05 PM
> > *To:* Yu Li; dev@hbase.apache.org<ma...@hbase.apache.org>
> > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> >
> > HI Yu
> >
> > I applied your path (the one that didn't have any V number) to 1.2.2
> > manually. I'm waiting to see the results. I should circle back if there
> are
> > issues.
> > ------------------------------
> > *From:* Yu Li <ca...@gmail.com>>
> > *Sent:* Sunday, March 5, 2017 7:06:27 AM
> > *To:* dev@hbase.apache.org<ma...@hbase.apache.org>
> > *Cc:* jeffsaremi@hotmail.com<ma...@hotmail.com>
> > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > @jeff normally it's recommended to try the latest one, say v5 in this
> > case.
> >
> > btw, one thing to make clear: only #4 in my list is missing in current
> > code base, all others are already there for usage.
> >
> > Best Regards,
> > Yu
> >
> > On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org>> wrote:
> >
> >> -user@hbase to bcc
> >> +dev@hbase
> >>
> >> Jeff,
> >>
> >> These kinds of questions get into things that we only handle within
> >> the development community. I've copied in the appropriate dev@hbase
> >> mailing list, please ensure you are subscribed there so you can see
> >> any further discussion.
> >>
> >> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>>
> >> wrote:
> >> > Yu
> >> > Of the patches attached to HBASE-15160, do I need to apply all (v2,
> v3,
> >> ...) or just HBASE-15160.patch ?
> >> > Also how would I know against what version this patch was created?
> >> > thanks
> >> >
> >> >
> >> >
> >> >
> >> > ________________________________
> >> > From: jeff saremi <je...@hotmail.com>>
> >> > Sent: Friday, March 3, 2017 10:34:00 AM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> stage of processing a request
> >> >
> >> > Thanks a lot Yu
> >> >
> >> > These are truly the metrics we care about about at this point. It is
> >> sad to see that such important metrics were removed from the code.
> >> >
> >> > I will try to apply your patch on my own to the version of HBase we
> >> have. We definitely need these.
> >> >
> >> > Other solutions like HTrace are not as urgent as having these few
> >> metrics you talked about here. So if we can get these merged with the
> code
> >> we should be happy.
> >> >
> >> >
> >> > ________________________________
> >> > From: Yu Li <ca...@gmail.com>>
> >> > Sent: Friday, March 3, 2017 9:54:29 AM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> stage of processing a request
> >> >
> >> > Hi Jeff,
> >> >
> >> > If the question is simply monitoring HDFS read/write latencies, please
> >> > refer to HBASE-15160 <https://issues.apache.org/
> jira/browse/HBASE-15160
> >> >,
> >> > there's a patch but not committed yet, and probably cannot apply
> >> cleanly on
> >> > current code base, but still some good reference IMHO, so JFYI.
> >> >
> >> > To get an overview of how quickly the system could respond and what
> >> might
> >> > be the root cause of the spikes, we only need to monitor the
> >> > average/p99/p999 latency of below metrics (stages):
> >> > 1. totalCallTime: time from request arriving at server to sending
> >> response
> >> > 2. processCallTime: time for the server to process the call,
> regardless
> >> of
> >> > the time this call being queued
> >> > 3. queueCallTime: time the call has been queued
> >> > 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> >> > HBASE-15160
> >> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
> >> request
> >> >
> >> > However, for your original question, that to monitor the whole trace
> of
> >> a
> >> > single request, I'm afraid no mature solution for the time being just
> as
> >> > Stack mentioned.
> >> >
> >> > Hope my answer helps (smile).
> >> >
> >> > Best Regards,
> >> > Yu
> >> >
> >> > On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com>> wrote:
> >> >
> >> >> anything would help. thanks
> >> >>
> >> >> ________________________________
> >> >> From: saint.ack@gmail.com<ma...@gmail.com> <sa...@gmail.com>> on behalf of Stack <
> >> >> stack@duboce.net<ma...@duboce.net>>
> >> >> Sent: Thursday, March 2, 2017 9:53:41 PM
> >> >> To: Hbase-User
> >> >> Subject: Re: Need guidance on getting detailed elapsed times in every
> >> >> stage of processing a request
> >> >>
> >> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
> >
> >> >> wrote:
> >> >>
> >> >> > So i'd like to come back to my original question on how to get
> about
> >> >> > separating the latency of HDFS from HBase.
> >> >> >
> >> >> >
> >> >> That is a simple question to which we do not have an answer
> >> unfortunately
> >> >> (we should). If interested, I could describe how you might do it. I
> >> don't
> >> >> think it would take much work.
> >> >>
> >> >> St.Ack
> >> >>
> >> >>
> >> >>
> >> >> > Is there a most appropriate log4j TRACE option that could print out
> >> this
> >> >> > information to the logs?
> >> >> > Thanks
> >> >> >
> >> >> > ________________________________
> >> >> > From: jeff saremi <je...@hotmail.com>>
> >> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
> >> >> > To: Hbase-User
> >> >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> >> > stage of processing a request
> >> >> >
> >> >> > Thanks so much for the advice! Looking forward to when Tracing gets
> >> >> picked
> >> >> > up again
> >> >> >
> >> >> > ________________________________
> >> >> > From: saint.ack@gmail.com<ma...@gmail.com> <sa...@gmail.com>> on behalf of
> Stack <
> >> >> > stack@duboce.net<ma...@duboce.net>>
> >> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
> >> >> > To: Hbase-User
> >> >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> >> > stage of processing a request
> >> >> >
> >> >> > HBase/HTrace integration once worked but has long since rotted.
> >> >> > Refactorings of internals without proper respect for trace
> >> connections is
> >> >> > the main culprit. Updates in htrace and hdfs that need attention
> >> >> > reconnecting spans, etc., is another. On top of this, zipkin
> project
> >> has
> >> >> > seen a burst of effort of late that would seem to offer much
> promise
> >> if
> >> >> > someone of us spent some time rejiggering how HTrace and Zipkin
> >> relate.
> >> >> >
> >> >> > I would not waste any time on trying to setup HTrace for HBase at
> >> least
> >> >> > until after HBASE-14451 goes in, an issue that has been put aside
> >> with a
> >> >> > while now. Sorry if you've burned time on this to date.
> >> >> >
> >> >> > Yours,
> >> >> > St.Ack
> >> >> >
> >> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <
> jeffsaremi@hotmail.com<ma...@hotmail.com>>
> >> >> > wrote:
> >> >> >
> >> >> > > Where would i seek help for issues revolving around HTrace and
> >> zipkin?
> >> >> > > Here? Because I have configured everything the way documentation
> >> said
> >> >> > but i
> >> >> > > see nothing in the zipkin server or in the logs. nothing at all
> >> >> > >
> >> >> > > ________________________________
> >> >> > > From: jeff saremi <je...@hotmail.com>>
> >> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> >> >> > > To: user@hbase.apache.org<ma...@hbase.apache.org>
> >> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> >> every
> >> >> > > stage of processing a request
> >> >> > >
> >> >> > > No I had not. but it looks like what i needed. Thanks Ted.
> >> >> > >
> >> >> > > I'll see if I have any more questions after reading this.
> >> >> > >
> >> >> > > ________________________________
> >> >> > > From: Ted Yu <yu...@gmail.com>>
> >> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> >> >> > > To: user@hbase.apache.org<ma...@hbase.apache.org>
> >> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> >> every
> >> >> > > stage of processing a request
> >> >> > >
> >> >> > > Have you looked at:
> >> >> > > http://hbase.apache.org/book.html#tracing
> >> >> > >
> >> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
> >> jeffsaremi@hotmail.com<ma...@hotmail.com>>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > I think we need to get detailed information from HBase
> >> RegionServer
> >> >> > logs
> >> >> > > > on how a request (read or write) is processed. Specifically
> >> >> speaking, i
> >> >> > > > need to know of say 100 ms time spent in processing a write,
> how
> >> much
> >> >> > of
> >> >> > > it
> >> >> > > > was spent waiting for the HDFS?
> >> >> > > > What is the most efficient way of enabling this in log4j
> >> properties?
> >> >> > Are
> >> >> > > > there better mechanisms to get this information?
> >> >> > > >
> >> >> > > > If I can get this in the log, then I can process the logs
> >> offline or
> >> >> in
> >> >> > > > neartime and mount some dashboards on the top.
> >> >> > > >
> >> >> > > > thanks
> >> >> > > >
> >> >> > > >
> >> >> > > > Jeff
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Yu Li <ca...@gmail.com>.
HBASE-15160 is done with great efforts from @Enis. You could check the
latest patch there and do a customized backport if needed. @jeff
Best Regards,
Yu
On 16 March 2017 at 02:38, Enis Söztutar <en...@gmail.com> wrote:
> If I remember from the jira, it got stuck on perf testing the different
> approaches of incrementing the histograms inline versus using a
> multi-producer single consumer queue (blockingQueue, or Disruptor, etc),
> and emitting the metrics inline and updating the histograms from a
> background thread. Which approach is better should be tested.
>
> Let's revive the jira.
> Enis
>
> On Tue, Mar 14, 2017 at 6:50 PM, Yu Li <ca...@gmail.com> wrote:
>
> > Let me check with @Enis in JIRA and get back to you (maybe days, due to
> > schedule) later.
> >
> > Best Regards,
> > Yu
> >
> > On 15 March 2017 at 05:57, jeff saremi <je...@hotmail.com> wrote:
> >
> > > What's involved in getting this change merged into the main branch?
> These
> > > 2 counters (fsReadLatency, fsWriteLatency) are super important to us
> > > understanding what goes on behind every request. These are the minimum
> we
> > > need to have especially in the absence of HTrace.
> > >
> > > I just checked the latest code and it does not seem to have these.
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> > >
> > > ------------------------------
> > > *From:* jeff saremi <je...@hotmail.com>
> > > *Sent:* Sunday, March 5, 2017 3:41:05 PM
> > > *To:* Yu Li; dev@hbase.apache.org
> > > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > >
> > > HI Yu
> > >
> > > I applied your path (the one that didn't have any V number) to 1.2.2
> > > manually. I'm waiting to see the results. I should circle back if there
> > are
> > > issues.
> > > ------------------------------
> > > *From:* Yu Li <ca...@gmail.com>
> > > *Sent:* Sunday, March 5, 2017 7:06:27 AM
> > > *To:* dev@hbase.apache.org
> > > *Cc:* jeffsaremi@hotmail.com
> > > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > @jeff normally it's recommended to try the latest one, say v5 in this
> > > case.
> > >
> > > btw, one thing to make clear: only #4 in my list is missing in current
> > > code base, all others are already there for usage.
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org> wrote:
> > >
> > >> -user@hbase to bcc
> > >> +dev@hbase
> > >>
> > >> Jeff,
> > >>
> > >> These kinds of questions get into things that we only handle within
> > >> the development community. I've copied in the appropriate dev@hbase
> > >> mailing list, please ensure you are subscribed there so you can see
> > >> any further discussion.
> > >>
> > >> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>
> > >> wrote:
> > >> > Yu
> > >> > Of the patches attached to HBASE-15160, do I need to apply all (v2,
> > v3,
> > >> ...) or just HBASE-15160.patch ?
> > >> > Also how would I know against what version this patch was created?
> > >> > thanks
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > ________________________________
> > >> > From: jeff saremi <je...@hotmail.com>
> > >> > Sent: Friday, March 3, 2017 10:34:00 AM
> > >> > To: Hbase-User
> > >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> > >> stage of processing a request
> > >> >
> > >> > Thanks a lot Yu
> > >> >
> > >> > These are truly the metrics we care about about at this point. It is
> > >> sad to see that such important metrics were removed from the code.
> > >> >
> > >> > I will try to apply your patch on my own to the version of HBase we
> > >> have. We definitely need these.
> > >> >
> > >> > Other solutions like HTrace are not as urgent as having these few
> > >> metrics you talked about here. So if we can get these merged with the
> > code
> > >> we should be happy.
> > >> >
> > >> >
> > >> > ________________________________
> > >> > From: Yu Li <ca...@gmail.com>
> > >> > Sent: Friday, March 3, 2017 9:54:29 AM
> > >> > To: Hbase-User
> > >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> > >> stage of processing a request
> > >> >
> > >> > Hi Jeff,
> > >> >
> > >> > If the question is simply monitoring HDFS read/write latencies,
> please
> > >> > refer to HBASE-15160 <https://issues.apache.org/
> > jira/browse/HBASE-15160
> > >> >,
> > >> > there's a patch but not committed yet, and probably cannot apply
> > >> cleanly on
> > >> > current code base, but still some good reference IMHO, so JFYI.
> > >> >
> > >> > To get an overview of how quickly the system could respond and what
> > >> might
> > >> > be the root cause of the spikes, we only need to monitor the
> > >> > average/p99/p999 latency of below metrics (stages):
> > >> > 1. totalCallTime: time from request arriving at server to sending
> > >> response
> > >> > 2. processCallTime: time for the server to process the call,
> > regardless
> > >> of
> > >> > the time this call being queued
> > >> > 3. queueCallTime: time the call has been queued
> > >> > 4. HDFS read/pread/write time: time of HFile reading/writing, added
> in
> > >> > HBASE-15160
> > >> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
> > >> request
> > >> >
> > >> > However, for your original question, that to monitor the whole trace
> > of
> > >> a
> > >> > single request, I'm afraid no mature solution for the time being
> just
> > as
> > >> > Stack mentioned.
> > >> >
> > >> > Hope my answer helps (smile).
> > >> >
> > >> > Best Regards,
> > >> > Yu
> > >> >
> > >> > On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com>
> wrote:
> > >> >
> > >> >> anything would help. thanks
> > >> >>
> > >> >> ________________________________
> > >> >> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of
> Stack <
> > >> >> stack@duboce.net>
> > >> >> Sent: Thursday, March 2, 2017 9:53:41 PM
> > >> >> To: Hbase-User
> > >> >> Subject: Re: Need guidance on getting detailed elapsed times in
> every
> > >> >> stage of processing a request
> > >> >>
> > >> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <
> jeffsaremi@hotmail.com
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > So i'd like to come back to my original question on how to get
> > about
> > >> >> > separating the latency of HDFS from HBase.
> > >> >> >
> > >> >> >
> > >> >> That is a simple question to which we do not have an answer
> > >> unfortunately
> > >> >> (we should). If interested, I could describe how you might do it. I
> > >> don't
> > >> >> think it would take much work.
> > >> >>
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >>
> > >> >> > Is there a most appropriate log4j TRACE option that could print
> out
> > >> this
> > >> >> > information to the logs?
> > >> >> > Thanks
> > >> >> >
> > >> >> > ________________________________
> > >> >> > From: jeff saremi <je...@hotmail.com>
> > >> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
> > >> >> > To: Hbase-User
> > >> >> > Subject: Re: Need guidance on getting detailed elapsed times in
> > every
> > >> >> > stage of processing a request
> > >> >> >
> > >> >> > Thanks so much for the advice! Looking forward to when Tracing
> gets
> > >> >> picked
> > >> >> > up again
> > >> >> >
> > >> >> > ________________________________
> > >> >> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of
> > Stack <
> > >> >> > stack@duboce.net>
> > >> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
> > >> >> > To: Hbase-User
> > >> >> > Subject: Re: Need guidance on getting detailed elapsed times in
> > every
> > >> >> > stage of processing a request
> > >> >> >
> > >> >> > HBase/HTrace integration once worked but has long since rotted.
> > >> >> > Refactorings of internals without proper respect for trace
> > >> connections is
> > >> >> > the main culprit. Updates in htrace and hdfs that need attention
> > >> >> > reconnecting spans, etc., is another. On top of this, zipkin
> > project
> > >> has
> > >> >> > seen a burst of effort of late that would seem to offer much
> > promise
> > >> if
> > >> >> > someone of us spent some time rejiggering how HTrace and Zipkin
> > >> relate.
> > >> >> >
> > >> >> > I would not waste any time on trying to setup HTrace for HBase at
> > >> least
> > >> >> > until after HBASE-14451 goes in, an issue that has been put aside
> > >> with a
> > >> >> > while now. Sorry if you've burned time on this to date.
> > >> >> >
> > >> >> > Yours,
> > >> >> > St.Ack
> > >> >> >
> > >> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <
> > jeffsaremi@hotmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Where would i seek help for issues revolving around HTrace and
> > >> zipkin?
> > >> >> > > Here? Because I have configured everything the way
> documentation
> > >> said
> > >> >> > but i
> > >> >> > > see nothing in the zipkin server or in the logs. nothing at all
> > >> >> > >
> > >> >> > > ________________________________
> > >> >> > > From: jeff saremi <je...@hotmail.com>
> > >> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > >> >> > > To: user@hbase.apache.org
> > >> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> > >> every
> > >> >> > > stage of processing a request
> > >> >> > >
> > >> >> > > No I had not. but it looks like what i needed. Thanks Ted.
> > >> >> > >
> > >> >> > > I'll see if I have any more questions after reading this.
> > >> >> > >
> > >> >> > > ________________________________
> > >> >> > > From: Ted Yu <yu...@gmail.com>
> > >> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > >> >> > > To: user@hbase.apache.org
> > >> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> > >> every
> > >> >> > > stage of processing a request
> > >> >> > >
> > >> >> > > Have you looked at:
> > >> >> > > http://hbase.apache.org/book.html#tracing
> > >> >> > >
> > >> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
> > >> jeffsaremi@hotmail.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > I think we need to get detailed information from HBase
> > >> RegionServer
> > >> >> > logs
> > >> >> > > > on how a request (read or write) is processed. Specifically
> > >> >> speaking, i
> > >> >> > > > need to know of say 100 ms time spent in processing a write,
> > how
> > >> much
> > >> >> > of
> > >> >> > > it
> > >> >> > > > was spent waiting for the HDFS?
> > >> >> > > > What is the most efficient way of enabling this in log4j
> > >> properties?
> > >> >> > Are
> > >> >> > > > there better mechanisms to get this information?
> > >> >> > > >
> > >> >> > > > If I can get this in the log, then I can process the logs
> > >> offline or
> > >> >> in
> > >> >> > > > neartime and mount some dashboards on the top.
> > >> >> > > >
> > >> >> > > > thanks
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > Jeff
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> > >
> > >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Enis Söztutar <en...@gmail.com>.
If I remember from the jira, it got stuck on perf testing the different
approaches of incrementing the histograms inline versus using a
multi-producer single consumer queue (blockingQueue, or Disruptor, etc),
and emitting the metrics inline and updating the histograms from a
background thread. Which approach is better should be tested.
Let's revive the jira.
Enis
On Tue, Mar 14, 2017 at 6:50 PM, Yu Li <ca...@gmail.com> wrote:
> Let me check with @Enis in JIRA and get back to you (maybe days, due to
> schedule) later.
>
> Best Regards,
> Yu
>
> On 15 March 2017 at 05:57, jeff saremi <je...@hotmail.com> wrote:
>
> > What's involved in getting this change merged into the main branch? These
> > 2 counters (fsReadLatency, fsWriteLatency) are super important to us
> > understanding what goes on behind every request. These are the minimum we
> > need to have especially in the absence of HTrace.
> >
> > I just checked the latest code and it does not seem to have these.
> >
> > thanks
> >
> > Jeff
> >
> >
> >
> > ------------------------------
> > *From:* jeff saremi <je...@hotmail.com>
> > *Sent:* Sunday, March 5, 2017 3:41:05 PM
> > *To:* Yu Li; dev@hbase.apache.org
> > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> >
> > HI Yu
> >
> > I applied your path (the one that didn't have any V number) to 1.2.2
> > manually. I'm waiting to see the results. I should circle back if there
> are
> > issues.
> > ------------------------------
> > *From:* Yu Li <ca...@gmail.com>
> > *Sent:* Sunday, March 5, 2017 7:06:27 AM
> > *To:* dev@hbase.apache.org
> > *Cc:* jeffsaremi@hotmail.com
> > *Subject:* Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > @jeff normally it's recommended to try the latest one, say v5 in this
> > case.
> >
> > btw, one thing to make clear: only #4 in my list is missing in current
> > code base, all others are already there for usage.
> >
> > Best Regards,
> > Yu
> >
> > On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org> wrote:
> >
> >> -user@hbase to bcc
> >> +dev@hbase
> >>
> >> Jeff,
> >>
> >> These kinds of questions get into things that we only handle within
> >> the development community. I've copied in the appropriate dev@hbase
> >> mailing list, please ensure you are subscribed there so you can see
> >> any further discussion.
> >>
> >> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>
> >> wrote:
> >> > Yu
> >> > Of the patches attached to HBASE-15160, do I need to apply all (v2,
> v3,
> >> ...) or just HBASE-15160.patch ?
> >> > Also how would I know against what version this patch was created?
> >> > thanks
> >> >
> >> >
> >> >
> >> >
> >> > ________________________________
> >> > From: jeff saremi <je...@hotmail.com>
> >> > Sent: Friday, March 3, 2017 10:34:00 AM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> stage of processing a request
> >> >
> >> > Thanks a lot Yu
> >> >
> >> > These are truly the metrics we care about about at this point. It is
> >> sad to see that such important metrics were removed from the code.
> >> >
> >> > I will try to apply your patch on my own to the version of HBase we
> >> have. We definitely need these.
> >> >
> >> > Other solutions like HTrace are not as urgent as having these few
> >> metrics you talked about here. So if we can get these merged with the
> code
> >> we should be happy.
> >> >
> >> >
> >> > ________________________________
> >> > From: Yu Li <ca...@gmail.com>
> >> > Sent: Friday, March 3, 2017 9:54:29 AM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> stage of processing a request
> >> >
> >> > Hi Jeff,
> >> >
> >> > If the question is simply monitoring HDFS read/write latencies, please
> >> > refer to HBASE-15160 <https://issues.apache.org/
> jira/browse/HBASE-15160
> >> >,
> >> > there's a patch but not committed yet, and probably cannot apply
> >> cleanly on
> >> > current code base, but still some good reference IMHO, so JFYI.
> >> >
> >> > To get an overview of how quickly the system could respond and what
> >> might
> >> > be the root cause of the spikes, we only need to monitor the
> >> > average/p99/p999 latency of below metrics (stages):
> >> > 1. totalCallTime: time from request arriving at server to sending
> >> response
> >> > 2. processCallTime: time for the server to process the call,
> regardless
> >> of
> >> > the time this call being queued
> >> > 3. queueCallTime: time the call has been queued
> >> > 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> >> > HBASE-15160
> >> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
> >> request
> >> >
> >> > However, for your original question, that to monitor the whole trace
> of
> >> a
> >> > single request, I'm afraid no mature solution for the time being just
> as
> >> > Stack mentioned.
> >> >
> >> > Hope my answer helps (smile).
> >> >
> >> > Best Regards,
> >> > Yu
> >> >
> >> > On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
> >> >
> >> >> anything would help. thanks
> >> >>
> >> >> ________________________________
> >> >> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> >> >> stack@duboce.net>
> >> >> Sent: Thursday, March 2, 2017 9:53:41 PM
> >> >> To: Hbase-User
> >> >> Subject: Re: Need guidance on getting detailed elapsed times in every
> >> >> stage of processing a request
> >> >>
> >> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <jeffsaremi@hotmail.com
> >
> >> >> wrote:
> >> >>
> >> >> > So i'd like to come back to my original question on how to get
> about
> >> >> > separating the latency of HDFS from HBase.
> >> >> >
> >> >> >
> >> >> That is a simple question to which we do not have an answer
> >> unfortunately
> >> >> (we should). If interested, I could describe how you might do it. I
> >> don't
> >> >> think it would take much work.
> >> >>
> >> >> St.Ack
> >> >>
> >> >>
> >> >>
> >> >> > Is there a most appropriate log4j TRACE option that could print out
> >> this
> >> >> > information to the logs?
> >> >> > Thanks
> >> >> >
> >> >> > ________________________________
> >> >> > From: jeff saremi <je...@hotmail.com>
> >> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
> >> >> > To: Hbase-User
> >> >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> >> > stage of processing a request
> >> >> >
> >> >> > Thanks so much for the advice! Looking forward to when Tracing gets
> >> >> picked
> >> >> > up again
> >> >> >
> >> >> > ________________________________
> >> >> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of
> Stack <
> >> >> > stack@duboce.net>
> >> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
> >> >> > To: Hbase-User
> >> >> > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> >> > stage of processing a request
> >> >> >
> >> >> > HBase/HTrace integration once worked but has long since rotted.
> >> >> > Refactorings of internals without proper respect for trace
> >> connections is
> >> >> > the main culprit. Updates in htrace and hdfs that need attention
> >> >> > reconnecting spans, etc., is another. On top of this, zipkin
> project
> >> has
> >> >> > seen a burst of effort of late that would seem to offer much
> promise
> >> if
> >> >> > someone of us spent some time rejiggering how HTrace and Zipkin
> >> relate.
> >> >> >
> >> >> > I would not waste any time on trying to setup HTrace for HBase at
> >> least
> >> >> > until after HBASE-14451 goes in, an issue that has been put aside
> >> with a
> >> >> > while now. Sorry if you've burned time on this to date.
> >> >> >
> >> >> > Yours,
> >> >> > St.Ack
> >> >> >
> >> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <
> jeffsaremi@hotmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Where would i seek help for issues revolving around HTrace and
> >> zipkin?
> >> >> > > Here? Because I have configured everything the way documentation
> >> said
> >> >> > but i
> >> >> > > see nothing in the zipkin server or in the logs. nothing at all
> >> >> > >
> >> >> > > ________________________________
> >> >> > > From: jeff saremi <je...@hotmail.com>
> >> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> >> >> > > To: user@hbase.apache.org
> >> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> >> every
> >> >> > > stage of processing a request
> >> >> > >
> >> >> > > No I had not. but it looks like what i needed. Thanks Ted.
> >> >> > >
> >> >> > > I'll see if I have any more questions after reading this.
> >> >> > >
> >> >> > > ________________________________
> >> >> > > From: Ted Yu <yu...@gmail.com>
> >> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> >> >> > > To: user@hbase.apache.org
> >> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> >> every
> >> >> > > stage of processing a request
> >> >> > >
> >> >> > > Have you looked at:
> >> >> > > http://hbase.apache.org/book.html#tracing
> >> >> > >
> >> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
> >> jeffsaremi@hotmail.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > I think we need to get detailed information from HBase
> >> RegionServer
> >> >> > logs
> >> >> > > > on how a request (read or write) is processed. Specifically
> >> >> speaking, i
> >> >> > > > need to know of say 100 ms time spent in processing a write,
> how
> >> much
> >> >> > of
> >> >> > > it
> >> >> > > > was spent waiting for the HDFS?
> >> >> > > > What is the most efficient way of enabling this in log4j
> >> properties?
> >> >> > Are
> >> >> > > > there better mechanisms to get this information?
> >> >> > > >
> >> >> > > > If I can get this in the log, then I can process the logs
> >> offline or
> >> >> in
> >> >> > > > neartime and mount some dashboards on the top.
> >> >> > > >
> >> >> > > > thanks
> >> >> > > >
> >> >> > > >
> >> >> > > > Jeff
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Yu Li <ca...@gmail.com>.
Let me check with @Enis in JIRA and get back to you (maybe days, due to
schedule) later.
Best Regards,
Yu
On 15 March 2017 at 05:57, jeff saremi <je...@hotmail.com> wrote:
> What's involved in getting this change merged into the main branch? These
> 2 counters (fsReadLatency, fsWriteLatency) are super important to us
> understanding what goes on behind every request. These are the minimum we
> need to have especially in the absence of HTrace.
>
> I just checked the latest code and it does not seem to have these.
>
> thanks
>
> Jeff
>
>
>
> ------------------------------
> *From:* jeff saremi <je...@hotmail.com>
> *Sent:* Sunday, March 5, 2017 3:41:05 PM
> *To:* Yu Li; dev@hbase.apache.org
> *Subject:* Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
>
> HI Yu
>
> I applied your path (the one that didn't have any V number) to 1.2.2
> manually. I'm waiting to see the results. I should circle back if there are
> issues.
> ------------------------------
> *From:* Yu Li <ca...@gmail.com>
> *Sent:* Sunday, March 5, 2017 7:06:27 AM
> *To:* dev@hbase.apache.org
> *Cc:* jeffsaremi@hotmail.com
> *Subject:* Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> @jeff normally it's recommended to try the latest one, say v5 in this
> case.
>
> btw, one thing to make clear: only #4 in my list is missing in current
> code base, all others are already there for usage.
>
> Best Regards,
> Yu
>
> On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org> wrote:
>
>> -user@hbase to bcc
>> +dev@hbase
>>
>> Jeff,
>>
>> These kinds of questions get into things that we only handle within
>> the development community. I've copied in the appropriate dev@hbase
>> mailing list, please ensure you are subscribed there so you can see
>> any further discussion.
>>
>> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>
>> wrote:
>> > Yu
>> > Of the patches attached to HBASE-15160, do I need to apply all (v2, v3,
>> ...) or just HBASE-15160.patch ?
>> > Also how would I know against what version this patch was created?
>> > thanks
>> >
>> >
>> >
>> >
>> > ________________________________
>> > From: jeff saremi <je...@hotmail.com>
>> > Sent: Friday, March 3, 2017 10:34:00 AM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>> >
>> > Thanks a lot Yu
>> >
>> > These are truly the metrics we care about about at this point. It is
>> sad to see that such important metrics were removed from the code.
>> >
>> > I will try to apply your patch on my own to the version of HBase we
>> have. We definitely need these.
>> >
>> > Other solutions like HTrace are not as urgent as having these few
>> metrics you talked about here. So if we can get these merged with the code
>> we should be happy.
>> >
>> >
>> > ________________________________
>> > From: Yu Li <ca...@gmail.com>
>> > Sent: Friday, March 3, 2017 9:54:29 AM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>> >
>> > Hi Jeff,
>> >
>> > If the question is simply monitoring HDFS read/write latencies, please
>> > refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160
>> >,
>> > there's a patch but not committed yet, and probably cannot apply
>> cleanly on
>> > current code base, but still some good reference IMHO, so JFYI.
>> >
>> > To get an overview of how quickly the system could respond and what
>> might
>> > be the root cause of the spikes, we only need to monitor the
>> > average/p99/p999 latency of below metrics (stages):
>> > 1. totalCallTime: time from request arriving at server to sending
>> response
>> > 2. processCallTime: time for the server to process the call, regardless
>> of
>> > the time this call being queued
>> > 3. queueCallTime: time the call has been queued
>> > 4. HDFS read/pread/write time: time of HFile reading/writing, added in
>> > HBASE-15160
>> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
>> request
>> >
>> > However, for your original question, that to monitor the whole trace of
>> a
>> > single request, I'm afraid no mature solution for the time being just as
>> > Stack mentioned.
>> >
>> > Hope my answer helps (smile).
>> >
>> > Best Regards,
>> > Yu
>> >
>> > On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
>> >
>> >> anything would help. thanks
>> >>
>> >> ________________________________
>> >> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
>> >> stack@duboce.net>
>> >> Sent: Thursday, March 2, 2017 9:53:41 PM
>> >> To: Hbase-User
>> >> Subject: Re: Need guidance on getting detailed elapsed times in every
>> >> stage of processing a request
>> >>
>> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
>> >> wrote:
>> >>
>> >> > So i'd like to come back to my original question on how to get about
>> >> > separating the latency of HDFS from HBase.
>> >> >
>> >> >
>> >> That is a simple question to which we do not have an answer
>> unfortunately
>> >> (we should). If interested, I could describe how you might do it. I
>> don't
>> >> think it would take much work.
>> >>
>> >> St.Ack
>> >>
>> >>
>> >>
>> >> > Is there a most appropriate log4j TRACE option that could print out
>> this
>> >> > information to the logs?
>> >> > Thanks
>> >> >
>> >> > ________________________________
>> >> > From: jeff saremi <je...@hotmail.com>
>> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
>> >> > To: Hbase-User
>> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> >> > stage of processing a request
>> >> >
>> >> > Thanks so much for the advice! Looking forward to when Tracing gets
>> >> picked
>> >> > up again
>> >> >
>> >> > ________________________________
>> >> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
>> >> > stack@duboce.net>
>> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
>> >> > To: Hbase-User
>> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> >> > stage of processing a request
>> >> >
>> >> > HBase/HTrace integration once worked but has long since rotted.
>> >> > Refactorings of internals without proper respect for trace
>> connections is
>> >> > the main culprit. Updates in htrace and hdfs that need attention
>> >> > reconnecting spans, etc., is another. On top of this, zipkin project
>> has
>> >> > seen a burst of effort of late that would seem to offer much promise
>> if
>> >> > someone of us spent some time rejiggering how HTrace and Zipkin
>> relate.
>> >> >
>> >> > I would not waste any time on trying to setup HTrace for HBase at
>> least
>> >> > until after HBASE-14451 goes in, an issue that has been put aside
>> with a
>> >> > while now. Sorry if you've burned time on this to date.
>> >> >
>> >> > Yours,
>> >> > St.Ack
>> >> >
>> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
>> >> > wrote:
>> >> >
>> >> > > Where would i seek help for issues revolving around HTrace and
>> zipkin?
>> >> > > Here? Because I have configured everything the way documentation
>> said
>> >> > but i
>> >> > > see nothing in the zipkin server or in the logs. nothing at all
>> >> > >
>> >> > > ________________________________
>> >> > > From: jeff saremi <je...@hotmail.com>
>> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
>> >> > > To: user@hbase.apache.org
>> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
>> every
>> >> > > stage of processing a request
>> >> > >
>> >> > > No I had not. but it looks like what i needed. Thanks Ted.
>> >> > >
>> >> > > I'll see if I have any more questions after reading this.
>> >> > >
>> >> > > ________________________________
>> >> > > From: Ted Yu <yu...@gmail.com>
>> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
>> >> > > To: user@hbase.apache.org
>> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
>> every
>> >> > > stage of processing a request
>> >> > >
>> >> > > Have you looked at:
>> >> > > http://hbase.apache.org/book.html#tracing
>> >> > >
>> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
>> jeffsaremi@hotmail.com>
>> >> > > wrote:
>> >> > >
>> >> > > > I think we need to get detailed information from HBase
>> RegionServer
>> >> > logs
>> >> > > > on how a request (read or write) is processed. Specifically
>> >> speaking, i
>> >> > > > need to know of say 100 ms time spent in processing a write, how
>> much
>> >> > of
>> >> > > it
>> >> > > > was spent waiting for the HDFS?
>> >> > > > What is the most efficient way of enabling this in log4j
>> properties?
>> >> > Are
>> >> > > > there better mechanisms to get this information?
>> >> > > >
>> >> > > > If I can get this in the log, then I can process the logs
>> offline or
>> >> in
>> >> > > > neartime and mount some dashboards on the top.
>> >> > > >
>> >> > > > thanks
>> >> > > >
>> >> > > >
>> >> > > > Jeff
>> >> > > >
>> >> > >
>> >> >
>> >>
>>
>
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
What's involved in getting this change merged into the main branch? These 2 counters (fsReadLatency, fsWriteLatency) are super important to us understanding what goes on behind every request. These are the minimum we need to have especially in the absence of HTrace.
I just checked the latest code and it does not seem to have these.
thanks
Jeff
________________________________
From: jeff saremi <je...@hotmail.com>
Sent: Sunday, March 5, 2017 3:41:05 PM
To: Yu Li; dev@hbase.apache.org
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
HI Yu
I applied your path (the one that didn't have any V number) to 1.2.2 manually. I'm waiting to see the results. I should circle back if there are issues.
________________________________
From: Yu Li <ca...@gmail.com>
Sent: Sunday, March 5, 2017 7:06:27 AM
To: dev@hbase.apache.org
Cc: jeffsaremi@hotmail.com
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
@jeff normally it's recommended to try the latest one, say v5 in this case.
btw, one thing to make clear: only #4 in my list is missing in current code base, all others are already there for usage.
Best Regards,
Yu
On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org>> wrote:
-user@hbase to bcc
+dev@hbase
Jeff,
These kinds of questions get into things that we only handle within
the development community. I've copied in the appropriate dev@hbase
mailing list, please ensure you are subscribed there so you can see
any further discussion.
On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>> wrote:
> Yu
> Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) or just HBASE-15160.patch ?
> Also how would I know against what version this patch was created?
> thanks
>
>
>
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>>
> Sent: Friday, March 3, 2017 10:34:00 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Thanks a lot Yu
>
> These are truly the metrics we care about about at this point. It is sad to see that such important metrics were removed from the code.
>
> I will try to apply your patch on my own to the version of HBase we have. We definitely need these.
>
> Other solutions like HTrace are not as urgent as having these few metrics you talked about here. So if we can get these merged with the code we should be happy.
>
>
> ________________________________
> From: Yu Li <ca...@gmail.com>>
> Sent: Friday, March 3, 2017 9:54:29 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Hi Jeff,
>
> If the question is simply monitoring HDFS read/write latencies, please
> refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
> there's a patch but not committed yet, and probably cannot apply cleanly on
> current code base, but still some good reference IMHO, so JFYI.
>
> To get an overview of how quickly the system could respond and what might
> be the root cause of the spikes, we only need to monitor the
> average/p99/p999 latency of below metrics (stages):
> 1. totalCallTime: time from request arriving at server to sending response
> 2. processCallTime: time for the server to process the call, regardless of
> the time this call being queued
> 3. queueCallTime: time the call has been queued
> 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> HBASE-15160
> 5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
>
> However, for your original question, that to monitor the whole trace of a
> single request, I'm afraid no mature solution for the time being just as
> Stack mentioned.
>
> Hope my answer helps (smile).
>
> Best Regards,
> Yu
>
> On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com>> wrote:
>
>> anything would help. thanks
>>
>> ________________________________
>> From: saint.ack@gmail.com<ma...@gmail.com> <sa...@gmail.com>> on behalf of Stack <
>> stack@duboce.net<ma...@duboce.net>>
>> Sent: Thursday, March 2, 2017 9:53:41 PM
>> To: Hbase-User
>> Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>>
>> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>>
>> wrote:
>>
>> > So i'd like to come back to my original question on how to get about
>> > separating the latency of HDFS from HBase.
>> >
>> >
>> That is a simple question to which we do not have an answer unfortunately
>> (we should). If interested, I could describe how you might do it. I don't
>> think it would take much work.
>>
>> St.Ack
>>
>>
>>
>> > Is there a most appropriate log4j TRACE option that could print out this
>> > information to the logs?
>> > Thanks
>> >
>> > ________________________________
>> > From: jeff saremi <je...@hotmail.com>>
>> > Sent: Thursday, March 2, 2017 12:45:59 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > Thanks so much for the advice! Looking forward to when Tracing gets
>> picked
>> > up again
>> >
>> > ________________________________
>> > From: saint.ack@gmail.com<ma...@gmail.com> <sa...@gmail.com>> on behalf of Stack <
>> > stack@duboce.net<ma...@duboce.net>>
>> > Sent: Thursday, March 2, 2017 12:17:35 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > HBase/HTrace integration once worked but has long since rotted.
>> > Refactorings of internals without proper respect for trace connections is
>> > the main culprit. Updates in htrace and hdfs that need attention
>> > reconnecting spans, etc., is another. On top of this, zipkin project has
>> > seen a burst of effort of late that would seem to offer much promise if
>> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
>> >
>> > I would not waste any time on trying to setup HTrace for HBase at least
>> > until after HBASE-14451 goes in, an issue that has been put aside with a
>> > while now. Sorry if you've burned time on this to date.
>> >
>> > Yours,
>> > St.Ack
>> >
>> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>>
>> > wrote:
>> >
>> > > Where would i seek help for issues revolving around HTrace and zipkin?
>> > > Here? Because I have configured everything the way documentation said
>> > but i
>> > > see nothing in the zipkin server or in the logs. nothing at all
>> > >
>> > > ________________________________
>> > > From: jeff saremi <je...@hotmail.com>>
>> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
>> > > To: user@hbase.apache.org<ma...@hbase.apache.org>
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > No I had not. but it looks like what i needed. Thanks Ted.
>> > >
>> > > I'll see if I have any more questions after reading this.
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yu...@gmail.com>>
>> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
>> > > To: user@hbase.apache.org<ma...@hbase.apache.org>
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > Have you looked at:
>> > > http://hbase.apache.org/book.html#tracing
>> > >
>> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>>
>> > > wrote:
>> > >
>> > > > I think we need to get detailed information from HBase RegionServer
>> > logs
>> > > > on how a request (read or write) is processed. Specifically
>> speaking, i
>> > > > need to know of say 100 ms time spent in processing a write, how much
>> > of
>> > > it
>> > > > was spent waiting for the HDFS?
>> > > > What is the most efficient way of enabling this in log4j properties?
>> > Are
>> > > > there better mechanisms to get this information?
>> > > >
>> > > > If I can get this in the log, then I can process the logs offline or
>> in
>> > > > neartime and mount some dashboards on the top.
>> > > >
>> > > > thanks
>> > > >
>> > > >
>> > > > Jeff
>> > > >
>> > >
>> >
>>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
HI Yu
I applied your path (the one that didn't have any V number) to 1.2.2 manually. I'm waiting to see the results. I should circle back if there are issues.
________________________________
From: Yu Li <ca...@gmail.com>
Sent: Sunday, March 5, 2017 7:06:27 AM
To: dev@hbase.apache.org
Cc: jeffsaremi@hotmail.com
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
@jeff normally it's recommended to try the latest one, say v5 in this case.
btw, one thing to make clear: only #4 in my list is missing in current code base, all others are already there for usage.
Best Regards,
Yu
On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org>> wrote:
-user@hbase to bcc
+dev@hbase
Jeff,
These kinds of questions get into things that we only handle within
the development community. I've copied in the appropriate dev@hbase
mailing list, please ensure you are subscribed there so you can see
any further discussion.
On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>> wrote:
> Yu
> Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) or just HBASE-15160.patch ?
> Also how would I know against what version this patch was created?
> thanks
>
>
>
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>>
> Sent: Friday, March 3, 2017 10:34:00 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Thanks a lot Yu
>
> These are truly the metrics we care about about at this point. It is sad to see that such important metrics were removed from the code.
>
> I will try to apply your patch on my own to the version of HBase we have. We definitely need these.
>
> Other solutions like HTrace are not as urgent as having these few metrics you talked about here. So if we can get these merged with the code we should be happy.
>
>
> ________________________________
> From: Yu Li <ca...@gmail.com>>
> Sent: Friday, March 3, 2017 9:54:29 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Hi Jeff,
>
> If the question is simply monitoring HDFS read/write latencies, please
> refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
> there's a patch but not committed yet, and probably cannot apply cleanly on
> current code base, but still some good reference IMHO, so JFYI.
>
> To get an overview of how quickly the system could respond and what might
> be the root cause of the spikes, we only need to monitor the
> average/p99/p999 latency of below metrics (stages):
> 1. totalCallTime: time from request arriving at server to sending response
> 2. processCallTime: time for the server to process the call, regardless of
> the time this call being queued
> 3. queueCallTime: time the call has been queued
> 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> HBASE-15160
> 5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
>
> However, for your original question, that to monitor the whole trace of a
> single request, I'm afraid no mature solution for the time being just as
> Stack mentioned.
>
> Hope my answer helps (smile).
>
> Best Regards,
> Yu
>
> On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com>> wrote:
>
>> anything would help. thanks
>>
>> ________________________________
>> From: saint.ack@gmail.com<ma...@gmail.com> <sa...@gmail.com>> on behalf of Stack <
>> stack@duboce.net<ma...@duboce.net>>
>> Sent: Thursday, March 2, 2017 9:53:41 PM
>> To: Hbase-User
>> Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>>
>> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>>
>> wrote:
>>
>> > So i'd like to come back to my original question on how to get about
>> > separating the latency of HDFS from HBase.
>> >
>> >
>> That is a simple question to which we do not have an answer unfortunately
>> (we should). If interested, I could describe how you might do it. I don't
>> think it would take much work.
>>
>> St.Ack
>>
>>
>>
>> > Is there a most appropriate log4j TRACE option that could print out this
>> > information to the logs?
>> > Thanks
>> >
>> > ________________________________
>> > From: jeff saremi <je...@hotmail.com>>
>> > Sent: Thursday, March 2, 2017 12:45:59 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > Thanks so much for the advice! Looking forward to when Tracing gets
>> picked
>> > up again
>> >
>> > ________________________________
>> > From: saint.ack@gmail.com<ma...@gmail.com> <sa...@gmail.com>> on behalf of Stack <
>> > stack@duboce.net<ma...@duboce.net>>
>> > Sent: Thursday, March 2, 2017 12:17:35 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > HBase/HTrace integration once worked but has long since rotted.
>> > Refactorings of internals without proper respect for trace connections is
>> > the main culprit. Updates in htrace and hdfs that need attention
>> > reconnecting spans, etc., is another. On top of this, zipkin project has
>> > seen a burst of effort of late that would seem to offer much promise if
>> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
>> >
>> > I would not waste any time on trying to setup HTrace for HBase at least
>> > until after HBASE-14451 goes in, an issue that has been put aside with a
>> > while now. Sorry if you've burned time on this to date.
>> >
>> > Yours,
>> > St.Ack
>> >
>> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>>
>> > wrote:
>> >
>> > > Where would i seek help for issues revolving around HTrace and zipkin?
>> > > Here? Because I have configured everything the way documentation said
>> > but i
>> > > see nothing in the zipkin server or in the logs. nothing at all
>> > >
>> > > ________________________________
>> > > From: jeff saremi <je...@hotmail.com>>
>> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
>> > > To: user@hbase.apache.org<ma...@hbase.apache.org>
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > No I had not. but it looks like what i needed. Thanks Ted.
>> > >
>> > > I'll see if I have any more questions after reading this.
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yu...@gmail.com>>
>> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
>> > > To: user@hbase.apache.org<ma...@hbase.apache.org>
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > Have you looked at:
>> > > http://hbase.apache.org/book.html#tracing
>> > >
>> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>>
>> > > wrote:
>> > >
>> > > > I think we need to get detailed information from HBase RegionServer
>> > logs
>> > > > on how a request (read or write) is processed. Specifically
>> speaking, i
>> > > > need to know of say 100 ms time spent in processing a write, how much
>> > of
>> > > it
>> > > > was spent waiting for the HDFS?
>> > > > What is the most efficient way of enabling this in log4j properties?
>> > Are
>> > > > there better mechanisms to get this information?
>> > > >
>> > > > If I can get this in the log, then I can process the logs offline or
>> in
>> > > > neartime and mount some dashboards on the top.
>> > > >
>> > > > thanks
>> > > >
>> > > >
>> > > > Jeff
>> > > >
>> > >
>> >
>>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Yu Li <ca...@gmail.com>.
@jeff normally it's recommended to try the latest one, say v5 in this case.
btw, one thing to make clear: only #4 in my list is missing in current code
base, all others are already there for usage.
Best Regards,
Yu
On 4 March 2017 at 04:02, Sean Busbey <bu...@apache.org> wrote:
> -user@hbase to bcc
> +dev@hbase
>
> Jeff,
>
> These kinds of questions get into things that we only handle within
> the development community. I've copied in the appropriate dev@hbase
> mailing list, please ensure you are subscribed there so you can see
> any further discussion.
>
> On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com>
> wrote:
> > Yu
> > Of the patches attached to HBASE-15160, do I need to apply all (v2, v3,
> ...) or just HBASE-15160.patch ?
> > Also how would I know against what version this patch was created?
> > thanks
> >
> >
> >
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Friday, March 3, 2017 10:34:00 AM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
> >
> > Thanks a lot Yu
> >
> > These are truly the metrics we care about about at this point. It is sad
> to see that such important metrics were removed from the code.
> >
> > I will try to apply your patch on my own to the version of HBase we
> have. We definitely need these.
> >
> > Other solutions like HTrace are not as urgent as having these few
> metrics you talked about here. So if we can get these merged with the code
> we should be happy.
> >
> >
> > ________________________________
> > From: Yu Li <ca...@gmail.com>
> > Sent: Friday, March 3, 2017 9:54:29 AM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
> >
> > Hi Jeff,
> >
> > If the question is simply monitoring HDFS read/write latencies, please
> > refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160
> >,
> > there's a patch but not committed yet, and probably cannot apply cleanly
> on
> > current code base, but still some good reference IMHO, so JFYI.
> >
> > To get an overview of how quickly the system could respond and what might
> > be the root cause of the spikes, we only need to monitor the
> > average/p99/p999 latency of below metrics (stages):
> > 1. totalCallTime: time from request arriving at server to sending
> response
> > 2. processCallTime: time for the server to process the call, regardless
> of
> > the time this call being queued
> > 3. queueCallTime: time the call has been queued
> > 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> > HBASE-15160
> > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing
> request
> >
> > However, for your original question, that to monitor the whole trace of a
> > single request, I'm afraid no mature solution for the time being just as
> > Stack mentioned.
> >
> > Hope my answer helps (smile).
> >
> > Best Regards,
> > Yu
> >
> > On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
> >
> >> anything would help. thanks
> >>
> >> ________________________________
> >> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> >> stack@duboce.net>
> >> Sent: Thursday, March 2, 2017 9:53:41 PM
> >> To: Hbase-User
> >> Subject: Re: Need guidance on getting detailed elapsed times in every
> >> stage of processing a request
> >>
> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
> >> wrote:
> >>
> >> > So i'd like to come back to my original question on how to get about
> >> > separating the latency of HDFS from HBase.
> >> >
> >> >
> >> That is a simple question to which we do not have an answer
> unfortunately
> >> (we should). If interested, I could describe how you might do it. I
> don't
> >> think it would take much work.
> >>
> >> St.Ack
> >>
> >>
> >>
> >> > Is there a most appropriate log4j TRACE option that could print out
> this
> >> > information to the logs?
> >> > Thanks
> >> >
> >> > ________________________________
> >> > From: jeff saremi <je...@hotmail.com>
> >> > Sent: Thursday, March 2, 2017 12:45:59 PM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> > stage of processing a request
> >> >
> >> > Thanks so much for the advice! Looking forward to when Tracing gets
> >> picked
> >> > up again
> >> >
> >> > ________________________________
> >> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> >> > stack@duboce.net>
> >> > Sent: Thursday, March 2, 2017 12:17:35 PM
> >> > To: Hbase-User
> >> > Subject: Re: Need guidance on getting detailed elapsed times in every
> >> > stage of processing a request
> >> >
> >> > HBase/HTrace integration once worked but has long since rotted.
> >> > Refactorings of internals without proper respect for trace
> connections is
> >> > the main culprit. Updates in htrace and hdfs that need attention
> >> > reconnecting spans, etc., is another. On top of this, zipkin project
> has
> >> > seen a burst of effort of late that would seem to offer much promise
> if
> >> > someone of us spent some time rejiggering how HTrace and Zipkin
> relate.
> >> >
> >> > I would not waste any time on trying to setup HTrace for HBase at
> least
> >> > until after HBASE-14451 goes in, an issue that has been put aside
> with a
> >> > while now. Sorry if you've burned time on this to date.
> >> >
> >> > Yours,
> >> > St.Ack
> >> >
> >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
> >> > wrote:
> >> >
> >> > > Where would i seek help for issues revolving around HTrace and
> zipkin?
> >> > > Here? Because I have configured everything the way documentation
> said
> >> > but i
> >> > > see nothing in the zipkin server or in the logs. nothing at all
> >> > >
> >> > > ________________________________
> >> > > From: jeff saremi <je...@hotmail.com>
> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> >> > > To: user@hbase.apache.org
> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> > > stage of processing a request
> >> > >
> >> > > No I had not. but it looks like what i needed. Thanks Ted.
> >> > >
> >> > > I'll see if I have any more questions after reading this.
> >> > >
> >> > > ________________________________
> >> > > From: Ted Yu <yu...@gmail.com>
> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> >> > > To: user@hbase.apache.org
> >> > > Subject: Re: Need guidance on getting detailed elapsed times in
> every
> >> > > stage of processing a request
> >> > >
> >> > > Have you looked at:
> >> > > http://hbase.apache.org/book.html#tracing
> >> > >
> >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <
> jeffsaremi@hotmail.com>
> >> > > wrote:
> >> > >
> >> > > > I think we need to get detailed information from HBase
> RegionServer
> >> > logs
> >> > > > on how a request (read or write) is processed. Specifically
> >> speaking, i
> >> > > > need to know of say 100 ms time spent in processing a write, how
> much
> >> > of
> >> > > it
> >> > > > was spent waiting for the HDFS?
> >> > > > What is the most efficient way of enabling this in log4j
> properties?
> >> > Are
> >> > > > there better mechanisms to get this information?
> >> > > >
> >> > > > If I can get this in the log, then I can process the logs offline
> or
> >> in
> >> > > > neartime and mount some dashboards on the top.
> >> > > >
> >> > > > thanks
> >> > > >
> >> > > >
> >> > > > Jeff
> >> > > >
> >> > >
> >> >
> >>
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Sean Busbey <bu...@apache.org>.
-user@hbase to bcc
+dev@hbase
Jeff,
These kinds of questions get into things that we only handle within
the development community. I've copied in the appropriate dev@hbase
mailing list, please ensure you are subscribed there so you can see
any further discussion.
On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com> wrote:
> Yu
> Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) or just HBASE-15160.patch ?
> Also how would I know against what version this patch was created?
> thanks
>
>
>
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Friday, March 3, 2017 10:34:00 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Thanks a lot Yu
>
> These are truly the metrics we care about about at this point. It is sad to see that such important metrics were removed from the code.
>
> I will try to apply your patch on my own to the version of HBase we have. We definitely need these.
>
> Other solutions like HTrace are not as urgent as having these few metrics you talked about here. So if we can get these merged with the code we should be happy.
>
>
> ________________________________
> From: Yu Li <ca...@gmail.com>
> Sent: Friday, March 3, 2017 9:54:29 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Hi Jeff,
>
> If the question is simply monitoring HDFS read/write latencies, please
> refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
> there's a patch but not committed yet, and probably cannot apply cleanly on
> current code base, but still some good reference IMHO, so JFYI.
>
> To get an overview of how quickly the system could respond and what might
> be the root cause of the spikes, we only need to monitor the
> average/p99/p999 latency of below metrics (stages):
> 1. totalCallTime: time from request arriving at server to sending response
> 2. processCallTime: time for the server to process the call, regardless of
> the time this call being queued
> 3. queueCallTime: time the call has been queued
> 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> HBASE-15160
> 5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
>
> However, for your original question, that to monitor the whole trace of a
> single request, I'm afraid no mature solution for the time being just as
> Stack mentioned.
>
> Hope my answer helps (smile).
>
> Best Regards,
> Yu
>
> On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
>
>> anything would help. thanks
>>
>> ________________________________
>> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
>> stack@duboce.net>
>> Sent: Thursday, March 2, 2017 9:53:41 PM
>> To: Hbase-User
>> Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>>
>> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
>> wrote:
>>
>> > So i'd like to come back to my original question on how to get about
>> > separating the latency of HDFS from HBase.
>> >
>> >
>> That is a simple question to which we do not have an answer unfortunately
>> (we should). If interested, I could describe how you might do it. I don't
>> think it would take much work.
>>
>> St.Ack
>>
>>
>>
>> > Is there a most appropriate log4j TRACE option that could print out this
>> > information to the logs?
>> > Thanks
>> >
>> > ________________________________
>> > From: jeff saremi <je...@hotmail.com>
>> > Sent: Thursday, March 2, 2017 12:45:59 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > Thanks so much for the advice! Looking forward to when Tracing gets
>> picked
>> > up again
>> >
>> > ________________________________
>> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
>> > stack@duboce.net>
>> > Sent: Thursday, March 2, 2017 12:17:35 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > HBase/HTrace integration once worked but has long since rotted.
>> > Refactorings of internals without proper respect for trace connections is
>> > the main culprit. Updates in htrace and hdfs that need attention
>> > reconnecting spans, etc., is another. On top of this, zipkin project has
>> > seen a burst of effort of late that would seem to offer much promise if
>> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
>> >
>> > I would not waste any time on trying to setup HTrace for HBase at least
>> > until after HBASE-14451 goes in, an issue that has been put aside with a
>> > while now. Sorry if you've burned time on this to date.
>> >
>> > Yours,
>> > St.Ack
>> >
>> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
>> > wrote:
>> >
>> > > Where would i seek help for issues revolving around HTrace and zipkin?
>> > > Here? Because I have configured everything the way documentation said
>> > but i
>> > > see nothing in the zipkin server or in the logs. nothing at all
>> > >
>> > > ________________________________
>> > > From: jeff saremi <je...@hotmail.com>
>> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
>> > > To: user@hbase.apache.org
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > No I had not. but it looks like what i needed. Thanks Ted.
>> > >
>> > > I'll see if I have any more questions after reading this.
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yu...@gmail.com>
>> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
>> > > To: user@hbase.apache.org
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > Have you looked at:
>> > > http://hbase.apache.org/book.html#tracing
>> > >
>> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
>> > > wrote:
>> > >
>> > > > I think we need to get detailed information from HBase RegionServer
>> > logs
>> > > > on how a request (read or write) is processed. Specifically
>> speaking, i
>> > > > need to know of say 100 ms time spent in processing a write, how much
>> > of
>> > > it
>> > > > was spent waiting for the HDFS?
>> > > > What is the most efficient way of enabling this in log4j properties?
>> > Are
>> > > > there better mechanisms to get this information?
>> > > >
>> > > > If I can get this in the log, then I can process the logs offline or
>> in
>> > > > neartime and mount some dashboards on the top.
>> > > >
>> > > > thanks
>> > > >
>> > > >
>> > > > Jeff
>> > > >
>> > >
>> >
>>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Sean Busbey <bu...@apache.org>.
-user@hbase to bcc
+dev@hbase
Jeff,
These kinds of questions get into things that we only handle within
the development community. I've copied in the appropriate dev@hbase
mailing list, please ensure you are subscribed there so you can see
any further discussion.
On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <je...@hotmail.com> wrote:
> Yu
> Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) or just HBASE-15160.patch ?
> Also how would I know against what version this patch was created?
> thanks
>
>
>
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Friday, March 3, 2017 10:34:00 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Thanks a lot Yu
>
> These are truly the metrics we care about about at this point. It is sad to see that such important metrics were removed from the code.
>
> I will try to apply your patch on my own to the version of HBase we have. We definitely need these.
>
> Other solutions like HTrace are not as urgent as having these few metrics you talked about here. So if we can get these merged with the code we should be happy.
>
>
> ________________________________
> From: Yu Li <ca...@gmail.com>
> Sent: Friday, March 3, 2017 9:54:29 AM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
>
> Hi Jeff,
>
> If the question is simply monitoring HDFS read/write latencies, please
> refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
> there's a patch but not committed yet, and probably cannot apply cleanly on
> current code base, but still some good reference IMHO, so JFYI.
>
> To get an overview of how quickly the system could respond and what might
> be the root cause of the spikes, we only need to monitor the
> average/p99/p999 latency of below metrics (stages):
> 1. totalCallTime: time from request arriving at server to sending response
> 2. processCallTime: time for the server to process the call, regardless of
> the time this call being queued
> 3. queueCallTime: time the call has been queued
> 4. HDFS read/pread/write time: time of HFile reading/writing, added in
> HBASE-15160
> 5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
>
> However, for your original question, that to monitor the whole trace of a
> single request, I'm afraid no mature solution for the time being just as
> Stack mentioned.
>
> Hope my answer helps (smile).
>
> Best Regards,
> Yu
>
> On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
>
>> anything would help. thanks
>>
>> ________________________________
>> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
>> stack@duboce.net>
>> Sent: Thursday, March 2, 2017 9:53:41 PM
>> To: Hbase-User
>> Subject: Re: Need guidance on getting detailed elapsed times in every
>> stage of processing a request
>>
>> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
>> wrote:
>>
>> > So i'd like to come back to my original question on how to get about
>> > separating the latency of HDFS from HBase.
>> >
>> >
>> That is a simple question to which we do not have an answer unfortunately
>> (we should). If interested, I could describe how you might do it. I don't
>> think it would take much work.
>>
>> St.Ack
>>
>>
>>
>> > Is there a most appropriate log4j TRACE option that could print out this
>> > information to the logs?
>> > Thanks
>> >
>> > ________________________________
>> > From: jeff saremi <je...@hotmail.com>
>> > Sent: Thursday, March 2, 2017 12:45:59 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > Thanks so much for the advice! Looking forward to when Tracing gets
>> picked
>> > up again
>> >
>> > ________________________________
>> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
>> > stack@duboce.net>
>> > Sent: Thursday, March 2, 2017 12:17:35 PM
>> > To: Hbase-User
>> > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > stage of processing a request
>> >
>> > HBase/HTrace integration once worked but has long since rotted.
>> > Refactorings of internals without proper respect for trace connections is
>> > the main culprit. Updates in htrace and hdfs that need attention
>> > reconnecting spans, etc., is another. On top of this, zipkin project has
>> > seen a burst of effort of late that would seem to offer much promise if
>> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
>> >
>> > I would not waste any time on trying to setup HTrace for HBase at least
>> > until after HBASE-14451 goes in, an issue that has been put aside with a
>> > while now. Sorry if you've burned time on this to date.
>> >
>> > Yours,
>> > St.Ack
>> >
>> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
>> > wrote:
>> >
>> > > Where would i seek help for issues revolving around HTrace and zipkin?
>> > > Here? Because I have configured everything the way documentation said
>> > but i
>> > > see nothing in the zipkin server or in the logs. nothing at all
>> > >
>> > > ________________________________
>> > > From: jeff saremi <je...@hotmail.com>
>> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
>> > > To: user@hbase.apache.org
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > No I had not. but it looks like what i needed. Thanks Ted.
>> > >
>> > > I'll see if I have any more questions after reading this.
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yu...@gmail.com>
>> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
>> > > To: user@hbase.apache.org
>> > > Subject: Re: Need guidance on getting detailed elapsed times in every
>> > > stage of processing a request
>> > >
>> > > Have you looked at:
>> > > http://hbase.apache.org/book.html#tracing
>> > >
>> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
>> > > wrote:
>> > >
>> > > > I think we need to get detailed information from HBase RegionServer
>> > logs
>> > > > on how a request (read or write) is processed. Specifically
>> speaking, i
>> > > > need to know of say 100 ms time spent in processing a write, how much
>> > of
>> > > it
>> > > > was spent waiting for the HDFS?
>> > > > What is the most efficient way of enabling this in log4j properties?
>> > Are
>> > > > there better mechanisms to get this information?
>> > > >
>> > > > If I can get this in the log, then I can process the logs offline or
>> in
>> > > > neartime and mount some dashboards on the top.
>> > > >
>> > > > thanks
>> > > >
>> > > >
>> > > > Jeff
>> > > >
>> > >
>> >
>>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
Yu
Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) or just HBASE-15160.patch ?
Also how would I know against what version this patch was created?
thanks
________________________________
From: jeff saremi <je...@hotmail.com>
Sent: Friday, March 3, 2017 10:34:00 AM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Thanks a lot Yu
These are truly the metrics we care about about at this point. It is sad to see that such important metrics were removed from the code.
I will try to apply your patch on my own to the version of HBase we have. We definitely need these.
Other solutions like HTrace are not as urgent as having these few metrics you talked about here. So if we can get these merged with the code we should be happy.
________________________________
From: Yu Li <ca...@gmail.com>
Sent: Friday, March 3, 2017 9:54:29 AM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Hi Jeff,
If the question is simply monitoring HDFS read/write latencies, please
refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
there's a patch but not committed yet, and probably cannot apply cleanly on
current code base, but still some good reference IMHO, so JFYI.
To get an overview of how quickly the system could respond and what might
be the root cause of the spikes, we only need to monitor the
average/p99/p999 latency of below metrics (stages):
1. totalCallTime: time from request arriving at server to sending response
2. processCallTime: time for the server to process the call, regardless of
the time this call being queued
3. queueCallTime: time the call has been queued
4. HDFS read/pread/write time: time of HFile reading/writing, added in
HBASE-15160
5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
However, for your original question, that to monitor the whole trace of a
single request, I'm afraid no mature solution for the time being just as
Stack mentioned.
Hope my answer helps (smile).
Best Regards,
Yu
On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
> anything would help. thanks
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Thursday, March 2, 2017 9:53:41 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > So i'd like to come back to my original question on how to get about
> > separating the latency of HDFS from HBase.
> >
> >
> That is a simple question to which we do not have an answer unfortunately
> (we should). If interested, I could describe how you might do it. I don't
> think it would take much work.
>
> St.Ack
>
>
>
> > Is there a most appropriate log4j TRACE option that could print out this
> > information to the logs?
> > Thanks
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Thursday, March 2, 2017 12:45:59 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > Thanks so much for the advice! Looking forward to when Tracing gets
> picked
> > up again
> >
> > ________________________________
> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> > stack@duboce.net>
> > Sent: Thursday, March 2, 2017 12:17:35 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > HBase/HTrace integration once worked but has long since rotted.
> > Refactorings of internals without proper respect for trace connections is
> > the main culprit. Updates in htrace and hdfs that need attention
> > reconnecting spans, etc., is another. On top of this, zipkin project has
> > seen a burst of effort of late that would seem to offer much promise if
> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
> >
> > I would not waste any time on trying to setup HTrace for HBase at least
> > until after HBASE-14451 goes in, an issue that has been put aside with a
> > while now. Sorry if you've burned time on this to date.
> >
> > Yours,
> > St.Ack
> >
> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > Where would i seek help for issues revolving around HTrace and zipkin?
> > > Here? Because I have configured everything the way documentation said
> > but i
> > > see nothing in the zipkin server or in the logs. nothing at all
> > >
> > > ________________________________
> > > From: jeff saremi <je...@hotmail.com>
> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > No I had not. but it looks like what i needed. Thanks Ted.
> > >
> > > I'll see if I have any more questions after reading this.
> > >
> > > ________________________________
> > > From: Ted Yu <yu...@gmail.com>
> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > Have you looked at:
> > > http://hbase.apache.org/book.html#tracing
> > >
> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> > > wrote:
> > >
> > > > I think we need to get detailed information from HBase RegionServer
> > logs
> > > > on how a request (read or write) is processed. Specifically
> speaking, i
> > > > need to know of say 100 ms time spent in processing a write, how much
> > of
> > > it
> > > > was spent waiting for the HDFS?
> > > > What is the most efficient way of enabling this in log4j properties?
> > Are
> > > > there better mechanisms to get this information?
> > > >
> > > > If I can get this in the log, then I can process the logs offline or
> in
> > > > neartime and mount some dashboards on the top.
> > > >
> > > > thanks
> > > >
> > > >
> > > > Jeff
> > > >
> > >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
Thanks a lot Yu
These are truly the metrics we care about about at this point. It is sad to see that such important metrics were removed from the code.
I will try to apply your patch on my own to the version of HBase we have. We definitely need these.
Other solutions like HTrace are not as urgent as having these few metrics you talked about here. So if we can get these merged with the code we should be happy.
________________________________
From: Yu Li <ca...@gmail.com>
Sent: Friday, March 3, 2017 9:54:29 AM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Hi Jeff,
If the question is simply monitoring HDFS read/write latencies, please
refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
there's a patch but not committed yet, and probably cannot apply cleanly on
current code base, but still some good reference IMHO, so JFYI.
To get an overview of how quickly the system could respond and what might
be the root cause of the spikes, we only need to monitor the
average/p99/p999 latency of below metrics (stages):
1. totalCallTime: time from request arriving at server to sending response
2. processCallTime: time for the server to process the call, regardless of
the time this call being queued
3. queueCallTime: time the call has been queued
4. HDFS read/pread/write time: time of HFile reading/writing, added in
HBASE-15160
5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
However, for your original question, that to monitor the whole trace of a
single request, I'm afraid no mature solution for the time being just as
Stack mentioned.
Hope my answer helps (smile).
Best Regards,
Yu
On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
> anything would help. thanks
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Thursday, March 2, 2017 9:53:41 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > So i'd like to come back to my original question on how to get about
> > separating the latency of HDFS from HBase.
> >
> >
> That is a simple question to which we do not have an answer unfortunately
> (we should). If interested, I could describe how you might do it. I don't
> think it would take much work.
>
> St.Ack
>
>
>
> > Is there a most appropriate log4j TRACE option that could print out this
> > information to the logs?
> > Thanks
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Thursday, March 2, 2017 12:45:59 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > Thanks so much for the advice! Looking forward to when Tracing gets
> picked
> > up again
> >
> > ________________________________
> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> > stack@duboce.net>
> > Sent: Thursday, March 2, 2017 12:17:35 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > HBase/HTrace integration once worked but has long since rotted.
> > Refactorings of internals without proper respect for trace connections is
> > the main culprit. Updates in htrace and hdfs that need attention
> > reconnecting spans, etc., is another. On top of this, zipkin project has
> > seen a burst of effort of late that would seem to offer much promise if
> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
> >
> > I would not waste any time on trying to setup HTrace for HBase at least
> > until after HBASE-14451 goes in, an issue that has been put aside with a
> > while now. Sorry if you've burned time on this to date.
> >
> > Yours,
> > St.Ack
> >
> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > Where would i seek help for issues revolving around HTrace and zipkin?
> > > Here? Because I have configured everything the way documentation said
> > but i
> > > see nothing in the zipkin server or in the logs. nothing at all
> > >
> > > ________________________________
> > > From: jeff saremi <je...@hotmail.com>
> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > No I had not. but it looks like what i needed. Thanks Ted.
> > >
> > > I'll see if I have any more questions after reading this.
> > >
> > > ________________________________
> > > From: Ted Yu <yu...@gmail.com>
> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > Have you looked at:
> > > http://hbase.apache.org/book.html#tracing
> > >
> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> > > wrote:
> > >
> > > > I think we need to get detailed information from HBase RegionServer
> > logs
> > > > on how a request (read or write) is processed. Specifically
> speaking, i
> > > > need to know of say 100 ms time spent in processing a write, how much
> > of
> > > it
> > > > was spent waiting for the HDFS?
> > > > What is the most efficient way of enabling this in log4j properties?
> > Are
> > > > there better mechanisms to get this information?
> > > >
> > > > If I can get this in the log, then I can process the logs offline or
> in
> > > > neartime and mount some dashboards on the top.
> > > >
> > > > thanks
> > > >
> > > >
> > > > Jeff
> > > >
> > >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Yu Li <ca...@gmail.com>.
Hi Jeff,
If the question is simply monitoring HDFS read/write latencies, please
refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>,
there's a patch but not committed yet, and probably cannot apply cleanly on
current code base, but still some good reference IMHO, so JFYI.
To get an overview of how quickly the system could respond and what might
be the root cause of the spikes, we only need to monitor the
average/p99/p999 latency of below metrics (stages):
1. totalCallTime: time from request arriving at server to sending response
2. processCallTime: time for the server to process the call, regardless of
the time this call being queued
3. queueCallTime: time the call has been queued
4. HDFS read/pread/write time: time of HFile reading/writing, added in
HBASE-15160
5. WAL sync time: time of WAL sync to HDFS, critical path of writing request
However, for your original question, that to monitor the whole trace of a
single request, I'm afraid no mature solution for the time being just as
Stack mentioned.
Hope my answer helps (smile).
Best Regards,
Yu
On 4 March 2017 at 00:48, jeff saremi <je...@hotmail.com> wrote:
> anything would help. thanks
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Thursday, March 2, 2017 9:53:41 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > So i'd like to come back to my original question on how to get about
> > separating the latency of HDFS from HBase.
> >
> >
> That is a simple question to which we do not have an answer unfortunately
> (we should). If interested, I could describe how you might do it. I don't
> think it would take much work.
>
> St.Ack
>
>
>
> > Is there a most appropriate log4j TRACE option that could print out this
> > information to the logs?
> > Thanks
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Thursday, March 2, 2017 12:45:59 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > Thanks so much for the advice! Looking forward to when Tracing gets
> picked
> > up again
> >
> > ________________________________
> > From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> > stack@duboce.net>
> > Sent: Thursday, March 2, 2017 12:17:35 PM
> > To: Hbase-User
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > HBase/HTrace integration once worked but has long since rotted.
> > Refactorings of internals without proper respect for trace connections is
> > the main culprit. Updates in htrace and hdfs that need attention
> > reconnecting spans, etc., is another. On top of this, zipkin project has
> > seen a burst of effort of late that would seem to offer much promise if
> > someone of us spent some time rejiggering how HTrace and Zipkin relate.
> >
> > I would not waste any time on trying to setup HTrace for HBase at least
> > until after HBASE-14451 goes in, an issue that has been put aside with a
> > while now. Sorry if you've burned time on this to date.
> >
> > Yours,
> > St.Ack
> >
> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > Where would i seek help for issues revolving around HTrace and zipkin?
> > > Here? Because I have configured everything the way documentation said
> > but i
> > > see nothing in the zipkin server or in the logs. nothing at all
> > >
> > > ________________________________
> > > From: jeff saremi <je...@hotmail.com>
> > > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > No I had not. but it looks like what i needed. Thanks Ted.
> > >
> > > I'll see if I have any more questions after reading this.
> > >
> > > ________________________________
> > > From: Ted Yu <yu...@gmail.com>
> > > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Need guidance on getting detailed elapsed times in every
> > > stage of processing a request
> > >
> > > Have you looked at:
> > > http://hbase.apache.org/book.html#tracing
> > >
> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> > > wrote:
> > >
> > > > I think we need to get detailed information from HBase RegionServer
> > logs
> > > > on how a request (read or write) is processed. Specifically
> speaking, i
> > > > need to know of say 100 ms time spent in processing a write, how much
> > of
> > > it
> > > > was spent waiting for the HDFS?
> > > > What is the most efficient way of enabling this in log4j properties?
> > Are
> > > > there better mechanisms to get this information?
> > > >
> > > > If I can get this in the log, then I can process the logs offline or
> in
> > > > neartime and mount some dashboards on the top.
> > > >
> > > > thanks
> > > >
> > > >
> > > > Jeff
> > > >
> > >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
anything would help. thanks
________________________________
From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <st...@duboce.net>
Sent: Thursday, March 2, 2017 9:53:41 PM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com> wrote:
> So i'd like to come back to my original question on how to get about
> separating the latency of HDFS from HBase.
>
>
That is a simple question to which we do not have an answer unfortunately
(we should). If interested, I could describe how you might do it. I don't
think it would take much work.
St.Ack
> Is there a most appropriate log4j TRACE option that could print out this
> information to the logs?
> Thanks
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Thursday, March 2, 2017 12:45:59 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> Thanks so much for the advice! Looking forward to when Tracing gets picked
> up again
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Thursday, March 2, 2017 12:17:35 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> HBase/HTrace integration once worked but has long since rotted.
> Refactorings of internals without proper respect for trace connections is
> the main culprit. Updates in htrace and hdfs that need attention
> reconnecting spans, etc., is another. On top of this, zipkin project has
> seen a burst of effort of late that would seem to offer much promise if
> someone of us spent some time rejiggering how HTrace and Zipkin relate.
>
> I would not waste any time on trying to setup HTrace for HBase at least
> until after HBASE-14451 goes in, an issue that has been put aside with a
> while now. Sorry if you've burned time on this to date.
>
> Yours,
> St.Ack
>
> On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > Where would i seek help for issues revolving around HTrace and zipkin?
> > Here? Because I have configured everything the way documentation said
> but i
> > see nothing in the zipkin server or in the logs. nothing at all
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > To: user@hbase.apache.org
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > No I had not. but it looks like what i needed. Thanks Ted.
> >
> > I'll see if I have any more questions after reading this.
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > To: user@hbase.apache.org
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > Have you looked at:
> > http://hbase.apache.org/book.html#tracing
> >
> > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > I think we need to get detailed information from HBase RegionServer
> logs
> > > on how a request (read or write) is processed. Specifically speaking, i
> > > need to know of say 100 ms time spent in processing a write, how much
> of
> > it
> > > was spent waiting for the HDFS?
> > > What is the most efficient way of enabling this in log4j properties?
> Are
> > > there better mechanisms to get this information?
> > >
> > > If I can get this in the log, then I can process the logs offline or in
> > > neartime and mount some dashboards on the top.
> > >
> > > thanks
> > >
> > >
> > > Jeff
> > >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Stack <st...@duboce.net>.
On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <je...@hotmail.com> wrote:
> So i'd like to come back to my original question on how to get about
> separating the latency of HDFS from HBase.
>
>
That is a simple question to which we do not have an answer unfortunately
(we should). If interested, I could describe how you might do it. I don't
think it would take much work.
St.Ack
> Is there a most appropriate log4j TRACE option that could print out this
> information to the logs?
> Thanks
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Thursday, March 2, 2017 12:45:59 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> Thanks so much for the advice! Looking forward to when Tracing gets picked
> up again
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Thursday, March 2, 2017 12:17:35 PM
> To: Hbase-User
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> HBase/HTrace integration once worked but has long since rotted.
> Refactorings of internals without proper respect for trace connections is
> the main culprit. Updates in htrace and hdfs that need attention
> reconnecting spans, etc., is another. On top of this, zipkin project has
> seen a burst of effort of late that would seem to offer much promise if
> someone of us spent some time rejiggering how HTrace and Zipkin relate.
>
> I would not waste any time on trying to setup HTrace for HBase at least
> until after HBASE-14451 goes in, an issue that has been put aside with a
> while now. Sorry if you've burned time on this to date.
>
> Yours,
> St.Ack
>
> On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > Where would i seek help for issues revolving around HTrace and zipkin?
> > Here? Because I have configured everything the way documentation said
> but i
> > see nothing in the zipkin server or in the logs. nothing at all
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Tuesday, February 28, 2017 12:52:32 PM
> > To: user@hbase.apache.org
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > No I had not. but it looks like what i needed. Thanks Ted.
> >
> > I'll see if I have any more questions after reading this.
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Tuesday, February 28, 2017 12:47:08 PM
> > To: user@hbase.apache.org
> > Subject: Re: Need guidance on getting detailed elapsed times in every
> > stage of processing a request
> >
> > Have you looked at:
> > http://hbase.apache.org/book.html#tracing
> >
> > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > I think we need to get detailed information from HBase RegionServer
> logs
> > > on how a request (read or write) is processed. Specifically speaking, i
> > > need to know of say 100 ms time spent in processing a write, how much
> of
> > it
> > > was spent waiting for the HDFS?
> > > What is the most efficient way of enabling this in log4j properties?
> Are
> > > there better mechanisms to get this information?
> > >
> > > If I can get this in the log, then I can process the logs offline or in
> > > neartime and mount some dashboards on the top.
> > >
> > > thanks
> > >
> > >
> > > Jeff
> > >
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
So i'd like to come back to my original question on how to get about separating the latency of HDFS from HBase.
Is there a most appropriate log4j TRACE option that could print out this information to the logs?
Thanks
________________________________
From: jeff saremi <je...@hotmail.com>
Sent: Thursday, March 2, 2017 12:45:59 PM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Thanks so much for the advice! Looking forward to when Tracing gets picked up again
________________________________
From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <st...@duboce.net>
Sent: Thursday, March 2, 2017 12:17:35 PM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
HBase/HTrace integration once worked but has long since rotted.
Refactorings of internals without proper respect for trace connections is
the main culprit. Updates in htrace and hdfs that need attention
reconnecting spans, etc., is another. On top of this, zipkin project has
seen a burst of effort of late that would seem to offer much promise if
someone of us spent some time rejiggering how HTrace and Zipkin relate.
I would not waste any time on trying to setup HTrace for HBase at least
until after HBASE-14451 goes in, an issue that has been put aside with a
while now. Sorry if you've burned time on this to date.
Yours,
St.Ack
On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com> wrote:
> Where would i seek help for issues revolving around HTrace and zipkin?
> Here? Because I have configured everything the way documentation said but i
> see nothing in the zipkin server or in the logs. nothing at all
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Tuesday, February 28, 2017 12:52:32 PM
> To: user@hbase.apache.org
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> No I had not. but it looks like what i needed. Thanks Ted.
>
> I'll see if I have any more questions after reading this.
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Tuesday, February 28, 2017 12:47:08 PM
> To: user@hbase.apache.org
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> Have you looked at:
> http://hbase.apache.org/book.html#tracing
>
> On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > I think we need to get detailed information from HBase RegionServer logs
> > on how a request (read or write) is processed. Specifically speaking, i
> > need to know of say 100 ms time spent in processing a write, how much of
> it
> > was spent waiting for the HDFS?
> > What is the most efficient way of enabling this in log4j properties? Are
> > there better mechanisms to get this information?
> >
> > If I can get this in the log, then I can process the logs offline or in
> > neartime and mount some dashboards on the top.
> >
> > thanks
> >
> >
> > Jeff
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
Thanks so much for the advice! Looking forward to when Tracing gets picked up again
________________________________
From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <st...@duboce.net>
Sent: Thursday, March 2, 2017 12:17:35 PM
To: Hbase-User
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
HBase/HTrace integration once worked but has long since rotted.
Refactorings of internals without proper respect for trace connections is
the main culprit. Updates in htrace and hdfs that need attention
reconnecting spans, etc., is another. On top of this, zipkin project has
seen a burst of effort of late that would seem to offer much promise if
someone of us spent some time rejiggering how HTrace and Zipkin relate.
I would not waste any time on trying to setup HTrace for HBase at least
until after HBASE-14451 goes in, an issue that has been put aside with a
while now. Sorry if you've burned time on this to date.
Yours,
St.Ack
On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com> wrote:
> Where would i seek help for issues revolving around HTrace and zipkin?
> Here? Because I have configured everything the way documentation said but i
> see nothing in the zipkin server or in the logs. nothing at all
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Tuesday, February 28, 2017 12:52:32 PM
> To: user@hbase.apache.org
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> No I had not. but it looks like what i needed. Thanks Ted.
>
> I'll see if I have any more questions after reading this.
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Tuesday, February 28, 2017 12:47:08 PM
> To: user@hbase.apache.org
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> Have you looked at:
> http://hbase.apache.org/book.html#tracing
>
> On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > I think we need to get detailed information from HBase RegionServer logs
> > on how a request (read or write) is processed. Specifically speaking, i
> > need to know of say 100 ms time spent in processing a write, how much of
> it
> > was spent waiting for the HDFS?
> > What is the most efficient way of enabling this in log4j properties? Are
> > there better mechanisms to get this information?
> >
> > If I can get this in the log, then I can process the logs offline or in
> > neartime and mount some dashboards on the top.
> >
> > thanks
> >
> >
> > Jeff
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Stack <st...@duboce.net>.
HBase/HTrace integration once worked but has long since rotted.
Refactorings of internals without proper respect for trace connections is
the main culprit. Updates in htrace and hdfs that need attention
reconnecting spans, etc., is another. On top of this, zipkin project has
seen a burst of effort of late that would seem to offer much promise if
someone of us spent some time rejiggering how HTrace and Zipkin relate.
I would not waste any time on trying to setup HTrace for HBase at least
until after HBASE-14451 goes in, an issue that has been put aside with a
while now. Sorry if you've burned time on this to date.
Yours,
St.Ack
On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <je...@hotmail.com> wrote:
> Where would i seek help for issues revolving around HTrace and zipkin?
> Here? Because I have configured everything the way documentation said but i
> see nothing in the zipkin server or in the logs. nothing at all
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Tuesday, February 28, 2017 12:52:32 PM
> To: user@hbase.apache.org
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> No I had not. but it looks like what i needed. Thanks Ted.
>
> I'll see if I have any more questions after reading this.
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Tuesday, February 28, 2017 12:47:08 PM
> To: user@hbase.apache.org
> Subject: Re: Need guidance on getting detailed elapsed times in every
> stage of processing a request
>
> Have you looked at:
> http://hbase.apache.org/book.html#tracing
>
> On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > I think we need to get detailed information from HBase RegionServer logs
> > on how a request (read or write) is processed. Specifically speaking, i
> > need to know of say 100 ms time spent in processing a write, how much of
> it
> > was spent waiting for the HDFS?
> > What is the most efficient way of enabling this in log4j properties? Are
> > there better mechanisms to get this information?
> >
> > If I can get this in the log, then I can process the logs offline or in
> > neartime and mount some dashboards on the top.
> >
> > thanks
> >
> >
> > Jeff
> >
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
Where would i seek help for issues revolving around HTrace and zipkin? Here? Because I have configured everything the way documentation said but i see nothing in the zipkin server or in the logs. nothing at all
________________________________
From: jeff saremi <je...@hotmail.com>
Sent: Tuesday, February 28, 2017 12:52:32 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
No I had not. but it looks like what i needed. Thanks Ted.
I'll see if I have any more questions after reading this.
________________________________
From: Ted Yu <yu...@gmail.com>
Sent: Tuesday, February 28, 2017 12:47:08 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Have you looked at:
http://hbase.apache.org/book.html#tracing
On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
wrote:
> I think we need to get detailed information from HBase RegionServer logs
> on how a request (read or write) is processed. Specifically speaking, i
> need to know of say 100 ms time spent in processing a write, how much of it
> was spent waiting for the HDFS?
> What is the most efficient way of enabling this in log4j properties? Are
> there better mechanisms to get this information?
>
> If I can get this in the log, then I can process the logs offline or in
> neartime and mount some dashboards on the top.
>
> thanks
>
>
> Jeff
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by jeff saremi <je...@hotmail.com>.
No I had not. but it looks like what i needed. Thanks Ted.
I'll see if I have any more questions after reading this.
________________________________
From: Ted Yu <yu...@gmail.com>
Sent: Tuesday, February 28, 2017 12:47:08 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on getting detailed elapsed times in every stage of processing a request
Have you looked at:
http://hbase.apache.org/book.html#tracing
On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
wrote:
> I think we need to get detailed information from HBase RegionServer logs
> on how a request (read or write) is processed. Specifically speaking, i
> need to know of say 100 ms time spent in processing a write, how much of it
> was spent waiting for the HDFS?
> What is the most efficient way of enabling this in log4j properties? Are
> there better mechanisms to get this information?
>
> If I can get this in the log, then I can process the logs offline or in
> neartime and mount some dashboards on the top.
>
> thanks
>
>
> Jeff
>
Re: Need guidance on getting detailed elapsed times in every stage of
processing a request
Posted by Ted Yu <yu...@gmail.com>.
Have you looked at:
http://hbase.apache.org/book.html#tracing
On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <je...@hotmail.com>
wrote:
> I think we need to get detailed information from HBase RegionServer logs
> on how a request (read or write) is processed. Specifically speaking, i
> need to know of say 100 ms time spent in processing a write, how much of it
> was spent waiting for the HDFS?
> What is the most efficient way of enabling this in log4j properties? Are
> there better mechanisms to get this information?
>
> If I can get this in the log, then I can process the logs offline or in
> neartime and mount some dashboards on the top.
>
> thanks
>
>
> Jeff
>