You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jack Levin <ma...@gmail.com> on 2011/04/09 21:22:07 UTC
rpc call logging
It would be super nice to have RPC query call logging per region
server. Sometimes we have 'hot' regions getting hit by users, and
this drives the CPU on a particular RS server up, except we have no
idea what table it might be. I could rig the tcpdump on port 60020,
and log things with AWK, but that would be super hacky. An insight
what traffic drives the cpu will allow us (or anyone else) model the
tables to user patterns accordingly.
thanks.
-Jack
Re: rpc call logging
Posted by Alexandre Jaquet <al...@gmail.com>.
doesn't need in each case to be stored in a table but a rpc inside an
operating system could simply stream and broadcast the content
2011/4/9 Jack Levin <ma...@gmail.com>
> It would be super nice to have RPC query call logging per region
> server. Sometimes we have 'hot' regions getting hit by users, and
> this drives the CPU on a particular RS server up, except we have no
> idea what table it might be. I could rig the tcpdump on port 60020,
> and log things with AWK, but that would be super hacky. An insight
> what traffic drives the cpu will allow us (or anyone else) model the
> tables to user patterns accordingly.
>
> thanks.
>
> -Jack
>
Re: rpc call logging
Posted by Andrew Purtell <ap...@apache.org>.
This sounds like HBASE-2014: https://issues.apache.org/jira/browse/HBASE-2014
BTW apologies for the weird English in that issue, it appears I cut and pasted a request from our China development center without sufficient editing.
- Andy
Re: rpc call logging
Posted by Jack Levin <ma...@gmail.com>.
Ok, but I can't exactly use it for debugging unless I have some sort
of a viewer - also, this is for writes, but mostly our issues are
reads.
-Jack
On Sun, Apr 10, 2011 at 4:58 PM, Stack <sa...@gmail.com> wrote:
> That's our wal log
>
>
>
> On Apr 10, 2011, at 13:46, Jack Levin <ma...@gmail.com> wrote:
>
>> I would love it to be exactly like mysqld.log where you save all your
>> queries there.
>>
>> -Jack
>>
>> On Sun, Apr 10, 2011 at 12:31 PM, Stack <st...@duboce.net> wrote:
>>> They don't enough detail when you enable metrics on rpc?
>>>
>>> It should be easy enough hacking this in. What would you like to see
>>> on the query Jack? There'd be whether a Get/Delete/Put, etc., then
>>> what? We'd have to give this logging a different name than that of
>>> the general regionserver class because you'd never want to run a
>>> production system with it on; you'd only want to enable it say via the
>>> UI for a short period of time, and then turn it off, because it'd be a
>>> drag on perf and it would make for reams of output (Consider server
>>> doing 1k hits a second, or in your case Jack, 100k hits a second --
>>> smile -- then we'd fill the disks in short order)
>>>
>>> St.Ack
>>>
>>>
>>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <ma...@gmail.com> wrote:
>>>> Well, counts would help a lot, but actual queries displayed even more
>>>> so... For example I just found a bug in our code that runs in infinite
>>>> loop across only specific pages that cause region server CPU to
>>>> overrun. It would be easier for us to ID the bugs if we had all
>>>> queries logged.
>>>>
>>>> -Jack
>>>>
>>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>>>> BTW 3647 isn't in 0.90.2
>>>>>
>>>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>>>> production, there would be hundreds of regions, it is not realistic to spot
>>>>> hot regions in the current display.
>>>>> Do we need to add capability to sort the regions on regionserver.jsp
>>>>> according to read/write request counts ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>>> You mean per-region Jack?
>>>>>>
>>>>>> These went into TRUNK recently:
>>>>>>
>>>>>> HBASE-3647 Distinguish read and write request count in region
>>>>>> (Ted Yu via Stack)
>>>>>> HBASE-3704 Show per region request count in table.jsp
>>>>>> (Ted Yu via Stack)
>>>>>>
>>>>>> Would they help?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
>>>>>>> It would be super nice to have RPC query call logging per region
>>>>>>> server. Sometimes we have 'hot' regions getting hit by users, and
>>>>>>> this drives the CPU on a particular RS server up, except we have no
>>>>>>> idea what table it might be. I could rig the tcpdump on port 60020,
>>>>>>> and log things with AWK, but that would be super hacky. An insight
>>>>>>> what traffic drives the cpu will allow us (or anyone else) model the
>>>>>>> tables to user patterns accordingly.
>>>>>>>
>>>>>>> thanks.
>>>>>>>
>>>>>>> -Jack
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: rpc call logging
Posted by Stack <sa...@gmail.com>.
That's our wal log
On Apr 10, 2011, at 13:46, Jack Levin <ma...@gmail.com> wrote:
> I would love it to be exactly like mysqld.log where you save all your
> queries there.
>
> -Jack
>
> On Sun, Apr 10, 2011 at 12:31 PM, Stack <st...@duboce.net> wrote:
>> They don't enough detail when you enable metrics on rpc?
>>
>> It should be easy enough hacking this in. What would you like to see
>> on the query Jack? There'd be whether a Get/Delete/Put, etc., then
>> what? We'd have to give this logging a different name than that of
>> the general regionserver class because you'd never want to run a
>> production system with it on; you'd only want to enable it say via the
>> UI for a short period of time, and then turn it off, because it'd be a
>> drag on perf and it would make for reams of output (Consider server
>> doing 1k hits a second, or in your case Jack, 100k hits a second --
>> smile -- then we'd fill the disks in short order)
>>
>> St.Ack
>>
>>
>> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <ma...@gmail.com> wrote:
>>> Well, counts would help a lot, but actual queries displayed even more
>>> so... For example I just found a bug in our code that runs in infinite
>>> loop across only specific pages that cause region server CPU to
>>> overrun. It would be easier for us to ID the bugs if we had all
>>> queries logged.
>>>
>>> -Jack
>>>
>>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <yu...@gmail.com> wrote:
>>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>>> BTW 3647 isn't in 0.90.2
>>>>
>>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>>> production, there would be hundreds of regions, it is not realistic to spot
>>>> hot regions in the current display.
>>>> Do we need to add capability to sort the regions on regionserver.jsp
>>>> according to read/write request counts ?
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>>> You mean per-region Jack?
>>>>>
>>>>> These went into TRUNK recently:
>>>>>
>>>>> HBASE-3647 Distinguish read and write request count in region
>>>>> (Ted Yu via Stack)
>>>>> HBASE-3704 Show per region request count in table.jsp
>>>>> (Ted Yu via Stack)
>>>>>
>>>>> Would they help?
>>>>>
>>>>> St.Ack
>>>>>
>>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
>>>>>> It would be super nice to have RPC query call logging per region
>>>>>> server. Sometimes we have 'hot' regions getting hit by users, and
>>>>>> this drives the CPU on a particular RS server up, except we have no
>>>>>> idea what table it might be. I could rig the tcpdump on port 60020,
>>>>>> and log things with AWK, but that would be super hacky. An insight
>>>>>> what traffic drives the cpu will allow us (or anyone else) model the
>>>>>> tables to user patterns accordingly.
>>>>>>
>>>>>> thanks.
>>>>>>
>>>>>> -Jack
>>>>>>
>>>>>
>>>>
>>>
>>
Re: rpc call logging
Posted by Jack Levin <ma...@gmail.com>.
I would love it to be exactly like mysqld.log where you save all your
queries there.
-Jack
On Sun, Apr 10, 2011 at 12:31 PM, Stack <st...@duboce.net> wrote:
> They don't enough detail when you enable metrics on rpc?
>
> It should be easy enough hacking this in. What would you like to see
> on the query Jack? There'd be whether a Get/Delete/Put, etc., then
> what? We'd have to give this logging a different name than that of
> the general regionserver class because you'd never want to run a
> production system with it on; you'd only want to enable it say via the
> UI for a short period of time, and then turn it off, because it'd be a
> drag on perf and it would make for reams of output (Consider server
> doing 1k hits a second, or in your case Jack, 100k hits a second --
> smile -- then we'd fill the disks in short order)
>
> St.Ack
>
>
> On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <ma...@gmail.com> wrote:
>> Well, counts would help a lot, but actual queries displayed even more
>> so... For example I just found a bug in our code that runs in infinite
>> loop across only specific pages that cause region server CPU to
>> overrun. It would be easier for us to ID the bugs if we had all
>> queries logged.
>>
>> -Jack
>>
>> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <yu...@gmail.com> wrote:
>>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>>> BTW 3647 isn't in 0.90.2
>>>
>>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>>> an unsorted order w.r.t. read/write request counts. For any region server in
>>> production, there would be hundreds of regions, it is not realistic to spot
>>> hot regions in the current display.
>>> Do we need to add capability to sort the regions on regionserver.jsp
>>> according to read/write request counts ?
>>>
>>> Cheers
>>>
>>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> You mean per-region Jack?
>>>>
>>>> These went into TRUNK recently:
>>>>
>>>> HBASE-3647 Distinguish read and write request count in region
>>>> (Ted Yu via Stack)
>>>> HBASE-3704 Show per region request count in table.jsp
>>>> (Ted Yu via Stack)
>>>>
>>>> Would they help?
>>>>
>>>> St.Ack
>>>>
>>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
>>>> > It would be super nice to have RPC query call logging per region
>>>> > server. Sometimes we have 'hot' regions getting hit by users, and
>>>> > this drives the CPU on a particular RS server up, except we have no
>>>> > idea what table it might be. I could rig the tcpdump on port 60020,
>>>> > and log things with AWK, but that would be super hacky. An insight
>>>> > what traffic drives the cpu will allow us (or anyone else) model the
>>>> > tables to user patterns accordingly.
>>>> >
>>>> > thanks.
>>>> >
>>>> > -Jack
>>>> >
>>>>
>>>
>>
>
Re: rpc call logging
Posted by Stack <st...@duboce.net>.
They don't enough detail when you enable metrics on rpc?
It should be easy enough hacking this in. What would you like to see
on the query Jack? There'd be whether a Get/Delete/Put, etc., then
what? We'd have to give this logging a different name than that of
the general regionserver class because you'd never want to run a
production system with it on; you'd only want to enable it say via the
UI for a short period of time, and then turn it off, because it'd be a
drag on perf and it would make for reams of output (Consider server
doing 1k hits a second, or in your case Jack, 100k hits a second --
smile -- then we'd fill the disks in short order)
St.Ack
On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <ma...@gmail.com> wrote:
> Well, counts would help a lot, but actual queries displayed even more
> so... For example I just found a bug in our code that runs in infinite
> loop across only specific pages that cause region server CPU to
> overrun. It would be easier for us to ID the bugs if we had all
> queries logged.
>
> -Jack
>
> On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <yu...@gmail.com> wrote:
>> I think HBASE-3647 was a good start but Jack asked might be a little more.
>> BTW 3647 isn't in 0.90.2
>>
>> Currently regionserver.jsp displays RegionServerMetrics for every region in
>> an unsorted order w.r.t. read/write request counts. For any region server in
>> production, there would be hundreds of regions, it is not realistic to spot
>> hot regions in the current display.
>> Do we need to add capability to sort the regions on regionserver.jsp
>> according to read/write request counts ?
>>
>> Cheers
>>
>> On Sat, Apr 9, 2011 at 1:59 PM, Stack <st...@duboce.net> wrote:
>>
>>> You mean per-region Jack?
>>>
>>> These went into TRUNK recently:
>>>
>>> HBASE-3647 Distinguish read and write request count in region
>>> (Ted Yu via Stack)
>>> HBASE-3704 Show per region request count in table.jsp
>>> (Ted Yu via Stack)
>>>
>>> Would they help?
>>>
>>> St.Ack
>>>
>>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
>>> > It would be super nice to have RPC query call logging per region
>>> > server. Sometimes we have 'hot' regions getting hit by users, and
>>> > this drives the CPU on a particular RS server up, except we have no
>>> > idea what table it might be. I could rig the tcpdump on port 60020,
>>> > and log things with AWK, but that would be super hacky. An insight
>>> > what traffic drives the cpu will allow us (or anyone else) model the
>>> > tables to user patterns accordingly.
>>> >
>>> > thanks.
>>> >
>>> > -Jack
>>> >
>>>
>>
>
Re: rpc call logging
Posted by Jack Levin <ma...@gmail.com>.
Well, counts would help a lot, but actual queries displayed even more
so... For example I just found a bug in our code that runs in infinite
loop across only specific pages that cause region server CPU to
overrun. It would be easier for us to ID the bugs if we had all
queries logged.
-Jack
On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <yu...@gmail.com> wrote:
> I think HBASE-3647 was a good start but Jack asked might be a little more.
> BTW 3647 isn't in 0.90.2
>
> Currently regionserver.jsp displays RegionServerMetrics for every region in
> an unsorted order w.r.t. read/write request counts. For any region server in
> production, there would be hundreds of regions, it is not realistic to spot
> hot regions in the current display.
> Do we need to add capability to sort the regions on regionserver.jsp
> according to read/write request counts ?
>
> Cheers
>
> On Sat, Apr 9, 2011 at 1:59 PM, Stack <st...@duboce.net> wrote:
>
>> You mean per-region Jack?
>>
>> These went into TRUNK recently:
>>
>> HBASE-3647 Distinguish read and write request count in region
>> (Ted Yu via Stack)
>> HBASE-3704 Show per region request count in table.jsp
>> (Ted Yu via Stack)
>>
>> Would they help?
>>
>> St.Ack
>>
>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
>> > It would be super nice to have RPC query call logging per region
>> > server. Sometimes we have 'hot' regions getting hit by users, and
>> > this drives the CPU on a particular RS server up, except we have no
>> > idea what table it might be. I could rig the tcpdump on port 60020,
>> > and log things with AWK, but that would be super hacky. An insight
>> > what traffic drives the cpu will allow us (or anyone else) model the
>> > tables to user patterns accordingly.
>> >
>> > thanks.
>> >
>> > -Jack
>> >
>>
>
Re: rpc call logging
Posted by Ted Yu <yu...@gmail.com>.
I think HBASE-3647 was a good start but Jack asked might be a little more.
BTW 3647 isn't in 0.90.2
Currently regionserver.jsp displays RegionServerMetrics for every region in
an unsorted order w.r.t. read/write request counts. For any region server in
production, there would be hundreds of regions, it is not realistic to spot
hot regions in the current display.
Do we need to add capability to sort the regions on regionserver.jsp
according to read/write request counts ?
Cheers
On Sat, Apr 9, 2011 at 1:59 PM, Stack <st...@duboce.net> wrote:
> You mean per-region Jack?
>
> These went into TRUNK recently:
>
> HBASE-3647 Distinguish read and write request count in region
> (Ted Yu via Stack)
> HBASE-3704 Show per region request count in table.jsp
> (Ted Yu via Stack)
>
> Would they help?
>
> St.Ack
>
> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
> > It would be super nice to have RPC query call logging per region
> > server. Sometimes we have 'hot' regions getting hit by users, and
> > this drives the CPU on a particular RS server up, except we have no
> > idea what table it might be. I could rig the tcpdump on port 60020,
> > and log things with AWK, but that would be super hacky. An insight
> > what traffic drives the cpu will allow us (or anyone else) model the
> > tables to user patterns accordingly.
> >
> > thanks.
> >
> > -Jack
> >
>
Re: rpc call logging
Posted by Stack <st...@duboce.net>.
You mean per-region Jack?
These went into TRUNK recently:
HBASE-3647 Distinguish read and write request count in region
(Ted Yu via Stack)
HBASE-3704 Show per region request count in table.jsp
(Ted Yu via Stack)
Would they help?
St.Ack
On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <ma...@gmail.com> wrote:
> It would be super nice to have RPC query call logging per region
> server. Sometimes we have 'hot' regions getting hit by users, and
> this drives the CPU on a particular RS server up, except we have no
> idea what table it might be. I could rig the tcpdump on port 60020,
> and log things with AWK, but that would be super hacky. An insight
> what traffic drives the cpu will allow us (or anyone else) model the
> tables to user patterns accordingly.
>
> thanks.
>
> -Jack
>