You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Stefan Groschupf <sg...@101tec.com> on 2010/12/28 05:07:34 UTC
Hadoop RPC call response post processing
Hi All,
I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire.
Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :(
Background:
Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases.
My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool.
A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput.
Any ideas?
Thanks,
Stefan
Re: Hadoop RPC call response post processing
Posted by Stefan Groschupf <sg...@101tec.com>.
Hi Todd,
Thanks for the feedback.
> Nope, it's not currently pluggable, nor do I think there's any compelling
> reason to make it pluggable.
Well, one could argue with an interceptor / filter it would be very easy to add compression or encryption to RPC.
But since the nutch days the code base was never architected in extendable or modular way.
> Perhaps you can hack something in a git branch, and if it has good results
> on something like NNBench it could be a general contribution?
Thanks - I pass on that offer. The days waiting a half year to get a patch into the codebase are behind me. :)
I think I will just replace hadoop RPC with netty.
Cheers,
Stefan
Re: Hadoop RPC call response post processing
Posted by Ted Dunning <td...@maprtech.com>.
Nah... this is hadoop RPC
On Tue, Dec 28, 2010 at 6:42 PM, Lance Norskog <go...@gmail.com> wrote:
> Are you connecting to this JVM with RMI? RMI does a very nasty thing
> with garbage collection: it forces a blocking collection every 60
> seconds. Really. You have to change this with a system property.
>
> On Tue, Dec 28, 2010 at 5:56 PM, Todd Lipcon <to...@cloudera.com> wrote:
> > On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <sg...@101tec.com> wrote:
> >
> >> Hi Todd,
> >> Right, that is the code I'm looking into. Though Responder is inner
> private
> >> class and is created "responder = new Responder();"
> >> It would be great if the Responder implementation could be configured.
> >> Do you have any idea how to overwrite the Responder?
> >>
> >
> > Nope, it's not currently pluggable, nor do I think there's any compelling
> > reason to make it pluggable. It's coupled quite tightly to the
> > implementation right now.
> >
> > Perhaps you can hack something in a git branch, and if it has good
> results
> > on something like NNBench it could be a general contribution?
> >
> > -Todd
> >
> >
> >> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
> >>
> >> > Hi Stefan,
> >> >
> >> > Sounds interesting.
> >> >
> >> > Maybe you're looking for o.a.h.ipc.Server$Responder?
> >> >
> >> > -Todd
> >> >
> >> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com>
> wrote:
> >> >
> >> >> Hi All,
> >> >> I'm browsing the RPC code since quite a while now trying to find any
> >> entry
> >> >> point / interceptor slot that allows me to handle a RPC call response
> >> >> writable after it was send over the wire.
> >> >> Does anybody has an idea how break into the RPC code from outside.
> All
> >> the
> >> >> interesting methods are private. :(
> >> >>
> >> >> Background:
> >> >> Heavy use of the RPC allocates hugh amount of Writable objects. We
> saw
> >> in
> >> >> multiple systems that the garbage collect can get so busy that the
> jvm
> >> >> almost freezes for seconds. Things like zookeeper sessions time out
> in
> >> that
> >> >> cases.
> >> >> My idea is to create an object pool for writables. Borrowing an
> object
> >> from
> >> >> the pool is simple since this happen in our custom code, though we do
> >> know
> >> >> when the writable return was send over the wire and can be returned
> into
> >> the
> >> >> pool.
> >> >> A dirty hack would be to overwrite the write(out) method in the
> >> writable,
> >> >> assuming that is the last thing done with the writable, though turns
> out
> >> >> that this method is called in other cases too, e.g. to measure
> >> throughput.
> >> >>
> >> >> Any ideas?
> >> >>
> >> >> Thanks,
> >> >> Stefan
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Todd Lipcon
> >> > Software Engineer, Cloudera
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Hadoop RPC call response post processing
Posted by Lance Norskog <go...@gmail.com>.
Are you connecting to this JVM with RMI? RMI does a very nasty thing
with garbage collection: it forces a blocking collection every 60
seconds. Really. You have to change this with a system property.
On Tue, Dec 28, 2010 at 5:56 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <sg...@101tec.com> wrote:
>
>> Hi Todd,
>> Right, that is the code I'm looking into. Though Responder is inner private
>> class and is created "responder = new Responder();"
>> It would be great if the Responder implementation could be configured.
>> Do you have any idea how to overwrite the Responder?
>>
>
> Nope, it's not currently pluggable, nor do I think there's any compelling
> reason to make it pluggable. It's coupled quite tightly to the
> implementation right now.
>
> Perhaps you can hack something in a git branch, and if it has good results
> on something like NNBench it could be a general contribution?
>
> -Todd
>
>
>> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
>>
>> > Hi Stefan,
>> >
>> > Sounds interesting.
>> >
>> > Maybe you're looking for o.a.h.ipc.Server$Responder?
>> >
>> > -Todd
>> >
>> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com> wrote:
>> >
>> >> Hi All,
>> >> I'm browsing the RPC code since quite a while now trying to find any
>> entry
>> >> point / interceptor slot that allows me to handle a RPC call response
>> >> writable after it was send over the wire.
>> >> Does anybody has an idea how break into the RPC code from outside. All
>> the
>> >> interesting methods are private. :(
>> >>
>> >> Background:
>> >> Heavy use of the RPC allocates hugh amount of Writable objects. We saw
>> in
>> >> multiple systems that the garbage collect can get so busy that the jvm
>> >> almost freezes for seconds. Things like zookeeper sessions time out in
>> that
>> >> cases.
>> >> My idea is to create an object pool for writables. Borrowing an object
>> from
>> >> the pool is simple since this happen in our custom code, though we do
>> know
>> >> when the writable return was send over the wire and can be returned into
>> the
>> >> pool.
>> >> A dirty hack would be to overwrite the write(out) method in the
>> writable,
>> >> assuming that is the last thing done with the writable, though turns out
>> >> that this method is called in other cases too, e.g. to measure
>> throughput.
>> >>
>> >> Any ideas?
>> >>
>> >> Thanks,
>> >> Stefan
>> >
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Lance Norskog
goksron@gmail.com
Re: Hadoop RPC call response post processing
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <sg...@101tec.com> wrote:
> Hi Todd,
> Right, that is the code I'm looking into. Though Responder is inner private
> class and is created "responder = new Responder();"
> It would be great if the Responder implementation could be configured.
> Do you have any idea how to overwrite the Responder?
>
Nope, it's not currently pluggable, nor do I think there's any compelling
reason to make it pluggable. It's coupled quite tightly to the
implementation right now.
Perhaps you can hack something in a git branch, and if it has good results
on something like NNBench it could be a general contribution?
-Todd
> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
>
> > Hi Stefan,
> >
> > Sounds interesting.
> >
> > Maybe you're looking for o.a.h.ipc.Server$Responder?
> >
> > -Todd
> >
> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com> wrote:
> >
> >> Hi All,
> >> I'm browsing the RPC code since quite a while now trying to find any
> entry
> >> point / interceptor slot that allows me to handle a RPC call response
> >> writable after it was send over the wire.
> >> Does anybody has an idea how break into the RPC code from outside. All
> the
> >> interesting methods are private. :(
> >>
> >> Background:
> >> Heavy use of the RPC allocates hugh amount of Writable objects. We saw
> in
> >> multiple systems that the garbage collect can get so busy that the jvm
> >> almost freezes for seconds. Things like zookeeper sessions time out in
> that
> >> cases.
> >> My idea is to create an object pool for writables. Borrowing an object
> from
> >> the pool is simple since this happen in our custom code, though we do
> know
> >> when the writable return was send over the wire and can be returned into
> the
> >> pool.
> >> A dirty hack would be to overwrite the write(out) method in the
> writable,
> >> assuming that is the last thing done with the writable, though turns out
> >> that this method is called in other cases too, e.g. to measure
> throughput.
> >>
> >> Any ideas?
> >>
> >> Thanks,
> >> Stefan
> >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Hadoop RPC call response post processing
Posted by Stefan Groschupf <sg...@101tec.com>.
Hi Todd,
Right, that is the code I'm looking into. Though Responder is inner private class and is created "responder = new Responder();"
It would be great if the Responder implementation could be configured.
Do you have any idea how to overwrite the Responder?
Thanks,
Stefan
On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
> Hi Stefan,
>
> Sounds interesting.
>
> Maybe you're looking for o.a.h.ipc.Server$Responder?
>
> -Todd
>
> On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com> wrote:
>
>> Hi All,
>> I'm browsing the RPC code since quite a while now trying to find any entry
>> point / interceptor slot that allows me to handle a RPC call response
>> writable after it was send over the wire.
>> Does anybody has an idea how break into the RPC code from outside. All the
>> interesting methods are private. :(
>>
>> Background:
>> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
>> multiple systems that the garbage collect can get so busy that the jvm
>> almost freezes for seconds. Things like zookeeper sessions time out in that
>> cases.
>> My idea is to create an object pool for writables. Borrowing an object from
>> the pool is simple since this happen in our custom code, though we do know
>> when the writable return was send over the wire and can be returned into the
>> pool.
>> A dirty hack would be to overwrite the write(out) method in the writable,
>> assuming that is the last thing done with the writable, though turns out
>> that this method is called in other cases too, e.g. to measure throughput.
>>
>> Any ideas?
>>
>> Thanks,
>> Stefan
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Hadoop RPC call response post processing
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Stefan,
Sounds interesting.
Maybe you're looking for o.a.h.ipc.Server$Responder?
-Todd
On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com> wrote:
> Hi All,
> I'm browsing the RPC code since quite a while now trying to find any entry
> point / interceptor slot that allows me to handle a RPC call response
> writable after it was send over the wire.
> Does anybody has an idea how break into the RPC code from outside. All the
> interesting methods are private. :(
>
> Background:
> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
> multiple systems that the garbage collect can get so busy that the jvm
> almost freezes for seconds. Things like zookeeper sessions time out in that
> cases.
> My idea is to create an object pool for writables. Borrowing an object from
> the pool is simple since this happen in our custom code, though we do know
> when the writable return was send over the wire and can be returned into the
> pool.
> A dirty hack would be to overwrite the write(out) method in the writable,
> assuming that is the last thing done with the writable, though turns out
> that this method is called in other cases too, e.g. to measure throughput.
>
> Any ideas?
>
> Thanks,
> Stefan
--
Todd Lipcon
Software Engineer, Cloudera
Re: Hadoop RPC call response post processing
Posted by Ted Dunning <td...@maprtech.com>.
Knowing the tenuring distribution will tell a lot about that exact issue.
Ephemeral collections take on average less than one instruction per
allocation and the allocation itself is generally only a single instruction.
For ephemeral garbage, it is extremely unlikely that you can beat that.
So the real question is whether you are actually creating so much garbage
that you are over-whelming the collector or whether the data is much longer
lived than it should be. *That* can cause lots of collection costs.
To tell how long data lives, you need to get the tenuring distribution:
-XX:+PrintTenuringDistribution Prints details about the tenuring
distribution to standard out. It can be used to show this threshold and the
ages of objects in the new generation. It is also useful for observing the
lifetime distribution of an application.
On Tue, Dec 28, 2010 at 11:59 AM, Stefan Groschupf <sg...@101tec.com> wrote:
> I don't think the problem is allocation but garbage collection.
>
Re: Hadoop RPC call response post processing
Posted by Stefan Groschupf <sg...@101tec.com>.
Hi Ted,
I don't think the problem is allocation but garbage collection.
When the gc kicks in everything freezes. Of course changing the gc algorithm helps a little.
Stefan
On Dec 27, 2010, at 11:21 PM, Ted Dunning wrote:
> I would be very surprised if allocation itself is the problem as opposed to
> good old fashioned excess copying.
>
> It is very hard to write an allocator faster than the java generational gc,
> especially if you are talking about objects that are ephemeral.
>
> Have you looked at the tenuring distribution?
>
> On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com> wrote:
>
>> Hi All,
>> I'm browsing the RPC code since quite a while now trying to find any entry
>> point / interceptor slot that allows me to handle a RPC call response
>> writable after it was send over the wire.
>> Does anybody has an idea how break into the RPC code from outside. All the
>> interesting methods are private. :(
>>
>> Background:
>> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
>> multiple systems that the garbage collect can get so busy that the jvm
>> almost freezes for seconds. Things like zookeeper sessions time out in that
>> cases.
>> My idea is to create an object pool for writables. Borrowing an object from
>> the pool is simple since this happen in our custom code, though we do know
>> when the writable return was send over the wire and can be returned into the
>> pool.
>> A dirty hack would be to overwrite the write(out) method in the writable,
>> assuming that is the last thing done with the writable, though turns out
>> that this method is called in other cases too, e.g. to measure throughput.
>>
>> Any ideas?
>>
>> Thanks,
>> Stefan
Re: Hadoop RPC call response post processing
Posted by Ted Dunning <td...@maprtech.com>.
I would be very surprised if allocation itself is the problem as opposed to
good old fashioned excess copying.
It is very hard to write an allocator faster than the java generational gc,
especially if you are talking about objects that are ephemeral.
Have you looked at the tenuring distribution?
On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg...@101tec.com> wrote:
> Hi All,
> I'm browsing the RPC code since quite a while now trying to find any entry
> point / interceptor slot that allows me to handle a RPC call response
> writable after it was send over the wire.
> Does anybody has an idea how break into the RPC code from outside. All the
> interesting methods are private. :(
>
> Background:
> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
> multiple systems that the garbage collect can get so busy that the jvm
> almost freezes for seconds. Things like zookeeper sessions time out in that
> cases.
> My idea is to create an object pool for writables. Borrowing an object from
> the pool is simple since this happen in our custom code, though we do know
> when the writable return was send over the wire and can be returned into the
> pool.
> A dirty hack would be to overwrite the write(out) method in the writable,
> assuming that is the last thing done with the writable, though turns out
> that this method is called in other cases too, e.g. to measure throughput.
>
> Any ideas?
>
> Thanks,
> Stefan