You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Matt Corgan <mc...@hotpads.com> on 2013/01/28 01:16:47 UTC
avro-rpc: running multiple threads on the netty server
Hi,
I'm trying to familiarize myself with avro-rpc by running and tweaking the
example at https://github.com/jbaldassari/Avro-RPC. By the way, thanks for
creating this example James.
I'm using the BidderTest.sendBidRequestWithCallback method and am trying to
get the server to process the requests in parallel, which is something i'll
need in my application. I use the DelayInjectingBidder with delay of 90ms
so i can watch what's happening on the server. By debugging the client, I
see that all requests do in fact make it to the server without blocking,
but watching the server log i see that each request is processed
sequentially. So it takes ~9s to process 100 messages.
Digging into the NettyServer constructor, I see the default ChannelFactory
uses Executors.newCachedThreadPool() for the Boss and Worker executors. I
tried overriding the worker with Executors.newFixedThreadPool(8) but didn't
notice any difference. Looking at the threads running in Eclipse Debug
mode, i see the following threads which confirms that there is only one
worker:
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
Thread [main] (Running)
Thread [ReaderThread] (Running)
Thread [New I/O server boss #2 ([id: 0x6a5dd151, /0:0:0:0:0:0:0:0:45633])]
(Running)
Thread [Avro NettyTransceiver Boss 1] (Running)
Thread [New I/O server worker #2-1] (Running)
Thread [New I/O client worker #1-1] (Running)
I also see that the NettyTransceiver has configurable thread pools, but I'm
not clear on what the Transceiver's role is (client side thing?), and
fiddling with its ChannelFactory is causing exceptions.
Is is possible that the behavior of the BidderTest client doesn't trigger
the multi-threaded server even though it's configured correctly, or that
the server processes all requests from a single client sequentially?
Sidenote: i also tried using avro version 1.7.3 with the same results,
although the above thread pools had different names.
I wonder if someone could shed some light on what the difference is between
the Server and Transceiver thread pools and how to configure them for
multi-threaded server processing.
Thanks,
Matt
Re: avro-rpc: running multiple threads on the netty server
Posted by Matt Corgan <mc...@hotpads.com>.
Thanks Philip, I plan to try the http approach as well for clients that
can't use Netty
..sent from my phone
On Jan 30, 2013 7:05 PM, "Philip Zeyliger" <ph...@cloudera.com> wrote:
> Matt,
>
> I misread your question. The multiple-thread approach is with the
> Jetty/HTTP mechanism, not the Netty one. Sorry 'bout the noise.
>
>
> On Wed, Jan 30, 2013 at 7:04 PM, Philip Zeyliger <ph...@cloudera.com>wrote:
>
>> Hi Matt,
>>
>> I do this with some frequency. The trick is to use Jetty manually to
>> create an appropriate jetty.Server, and then start it. The snippets below
>> should help.
>>
>> -- Philip
>>
>> import javax.servlet.Servlet;
>>
>> import org.apache.avro.ipc.ResponderServlet;
>> import org.apache.avro.ipc.specific.SpecificResponder;
>> import org.mortbay.jetty.Server;
>> import org.mortbay.jetty.servlet.Context;
>> import org.mortbay.jetty.servlet.ServletHolder;
>> import org.mortbay.thread.QueuedThreadPool;
>>
>> public static Server createAvroServer(int port, String name, int
>> maxThreads,
>> int maxIdleTimeMs,
>> SpecificResponder responder)
>> throws EnterpriseServiceException
>> {
>> try {
>> Server httpServer = new Server(port);
>> QueuedThreadPool qtp = new QueuedThreadPool();
>> // QueuedThreadPool is jetty's thread pool implementation;
>> // this lets us give it a name.
>> qtp.setName(name);
>> qtp.setDaemon(true);
>> qtp.setMaxThreads(maxThreads);
>> qtp.setMaxIdleTimeMs(maxIdleTimeMs);
>> httpServer.setThreadPool(qtp);
>> Servlet servlet = new ResponderServlet(responder);
>> new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
>> "/*");
>> return httpServer;
>> } catch (Exception e) {
>> throw new EnterpriseServiceException(e);
>> }
>> }
>>
>> // main:
>> SpecificResponder responder = new SpecificResponder(
>> XXX.class,
>> new XXXImpl());
>> server = createAvroServer(listenPort, "somename",
>> maxThreads, maxIdleTimeMs, responder);
>> server.start();
>>
>>
>> On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>
>>> hmm - i haven't heard any suggestions in the past few days... maybe a
>>> really short version of the question: Can an Avro-rpc Netty server process
>>> requests in parallel? Will it still process in parallel if they're from
>>> the same client?
>>>
>>> Thanks much,
>>> Matt
>>>
>>>
>>> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <mc...@hotpads.com>wrote:
>>>
>>>> Hi,
>>>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>>>> the example at https://github.com/jbaldassari/Avro-RPC. By the way,
>>>> thanks for creating this example James.
>>>>
>>>> I'm using the BidderTest.sendBidRequestWithCallback method and am
>>>> trying to get the server to process the requests in parallel, which is
>>>> something i'll need in my application. I use the DelayInjectingBidder with
>>>> delay of 90ms so i can watch what's happening on the server. By debugging
>>>> the client, I see that all requests do in fact make it to the server
>>>> without blocking, but watching the server log i see that each request is
>>>> processed sequentially. So it takes ~9s to process 100 messages.
>>>>
>>>> Digging into the NettyServer constructor, I see the default
>>>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>>>> executors. I tried overriding the worker with
>>>> Executors.newFixedThreadPool(8) but didn't notice any difference. Looking
>>>> at the threads running in Eclipse Debug mode, i see the following threads
>>>> which confirms that there is only one worker:
>>>>
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at
>>>> localhost:50255
>>>> Thread [main] (Running)
>>>> Thread [ReaderThread] (Running)
>>>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,
>>>> /0:0:0:0:0:0:0:0:45633])] (Running)
>>>> Thread [Avro NettyTransceiver Boss 1] (Running)
>>>> Thread [New I/O server worker #2-1] (Running)
>>>> Thread [New I/O client worker #1-1] (Running)
>>>>
>>>> I also see that the NettyTransceiver has configurable thread pools, but
>>>> I'm not clear on what the Transceiver's role is (client side thing?), and
>>>> fiddling with its ChannelFactory is causing exceptions.
>>>>
>>>> Is is possible that the behavior of the BidderTest client doesn't
>>>> trigger the multi-threaded server even though it's configured correctly, or
>>>> that the server processes all requests from a single client sequentially?
>>>>
>>>> Sidenote: i also tried using avro version 1.7.3 with the same results,
>>>> although the above thread pools had different names.
>>>>
>>>> I wonder if someone could shed some light on what the difference is
>>>> between the Server and Transceiver thread pools and how to configure them
>>>> for multi-threaded server processing.
>>>>
>>>> Thanks,
>>>> Matt
>>>>
>>>>
>>>
>>
>
Re: avro-rpc: running multiple threads on the netty server
Posted by Philip Zeyliger <ph...@cloudera.com>.
Matt,
I misread your question. The multiple-thread approach is with the
Jetty/HTTP mechanism, not the Netty one. Sorry 'bout the noise.
On Wed, Jan 30, 2013 at 7:04 PM, Philip Zeyliger <ph...@cloudera.com>wrote:
> Hi Matt,
>
> I do this with some frequency. The trick is to use Jetty manually to
> create an appropriate jetty.Server, and then start it. The snippets below
> should help.
>
> -- Philip
>
> import javax.servlet.Servlet;
>
> import org.apache.avro.ipc.ResponderServlet;
> import org.apache.avro.ipc.specific.SpecificResponder;
> import org.mortbay.jetty.Server;
> import org.mortbay.jetty.servlet.Context;
> import org.mortbay.jetty.servlet.ServletHolder;
> import org.mortbay.thread.QueuedThreadPool;
>
> public static Server createAvroServer(int port, String name, int
> maxThreads,
> int maxIdleTimeMs,
> SpecificResponder responder)
> throws EnterpriseServiceException {
> try {
> Server httpServer = new Server(port);
> QueuedThreadPool qtp = new QueuedThreadPool();
> // QueuedThreadPool is jetty's thread pool implementation;
> // this lets us give it a name.
> qtp.setName(name);
> qtp.setDaemon(true);
> qtp.setMaxThreads(maxThreads);
> qtp.setMaxIdleTimeMs(maxIdleTimeMs);
> httpServer.setThreadPool(qtp);
> Servlet servlet = new ResponderServlet(responder);
> new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
> "/*");
> return httpServer;
> } catch (Exception e) {
> throw new EnterpriseServiceException(e);
> }
> }
>
> // main:
> SpecificResponder responder = new SpecificResponder(
> XXX.class,
> new XXXImpl());
> server = createAvroServer(listenPort, "somename",
> maxThreads, maxIdleTimeMs, responder);
> server.start();
>
>
> On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
>> hmm - i haven't heard any suggestions in the past few days... maybe a
>> really short version of the question: Can an Avro-rpc Netty server process
>> requests in parallel? Will it still process in parallel if they're from
>> the same client?
>>
>> Thanks much,
>> Matt
>>
>>
>> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>
>>> Hi,
>>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>>> the example at https://github.com/jbaldassari/Avro-RPC. By the way,
>>> thanks for creating this example James.
>>>
>>> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
>>> to get the server to process the requests in parallel, which is something
>>> i'll need in my application. I use the DelayInjectingBidder with delay of
>>> 90ms so i can watch what's happening on the server. By debugging the
>>> client, I see that all requests do in fact make it to the server without
>>> blocking, but watching the server log i see that each request is processed
>>> sequentially. So it takes ~9s to process 100 messages.
>>>
>>> Digging into the NettyServer constructor, I see the default
>>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>>> executors. I tried overriding the worker with
>>> Executors.newFixedThreadPool(8) but didn't notice any difference. Looking
>>> at the threads running in Eclipse Debug mode, i see the following threads
>>> which confirms that there is only one worker:
>>>
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
>>> Thread [main] (Running)
>>> Thread [ReaderThread] (Running)
>>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,
>>> /0:0:0:0:0:0:0:0:45633])] (Running)
>>> Thread [Avro NettyTransceiver Boss 1] (Running)
>>> Thread [New I/O server worker #2-1] (Running)
>>> Thread [New I/O client worker #1-1] (Running)
>>>
>>> I also see that the NettyTransceiver has configurable thread pools, but
>>> I'm not clear on what the Transceiver's role is (client side thing?), and
>>> fiddling with its ChannelFactory is causing exceptions.
>>>
>>> Is is possible that the behavior of the BidderTest client doesn't
>>> trigger the multi-threaded server even though it's configured correctly, or
>>> that the server processes all requests from a single client sequentially?
>>>
>>> Sidenote: i also tried using avro version 1.7.3 with the same results,
>>> although the above thread pools had different names.
>>>
>>> I wonder if someone could shed some light on what the difference is
>>> between the Server and Transceiver thread pools and how to configure them
>>> for multi-threaded server processing.
>>>
>>> Thanks,
>>> Matt
>>>
>>>
>>
>
Re: avro-rpc: running multiple threads on the netty server
Posted by Philip Zeyliger <ph...@cloudera.com>.
Hi Matt,
I do this with some frequency. The trick is to use Jetty manually to
create an appropriate jetty.Server, and then start it. The snippets below
should help.
-- Philip
import javax.servlet.Servlet;
import org.apache.avro.ipc.ResponderServlet;
import org.apache.avro.ipc.specific.SpecificResponder;
import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.jetty.servlet.ServletHolder;
import org.mortbay.thread.QueuedThreadPool;
public static Server createAvroServer(int port, String name, int
maxThreads,
int maxIdleTimeMs,
SpecificResponder responder)
throws EnterpriseServiceException {
try {
Server httpServer = new Server(port);
QueuedThreadPool qtp = new QueuedThreadPool();
// QueuedThreadPool is jetty's thread pool implementation;
// this lets us give it a name.
qtp.setName(name);
qtp.setDaemon(true);
qtp.setMaxThreads(maxThreads);
qtp.setMaxIdleTimeMs(maxIdleTimeMs);
httpServer.setThreadPool(qtp);
Servlet servlet = new ResponderServlet(responder);
new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
"/*");
return httpServer;
} catch (Exception e) {
throw new EnterpriseServiceException(e);
}
}
// main:
SpecificResponder responder = new SpecificResponder(
XXX.class,
new XXXImpl());
server = createAvroServer(listenPort, "somename",
maxThreads, maxIdleTimeMs, responder);
server.start();
On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <mc...@hotpads.com> wrote:
> hmm - i haven't heard any suggestions in the past few days... maybe a
> really short version of the question: Can an Avro-rpc Netty server process
> requests in parallel? Will it still process in parallel if they're from
> the same client?
>
> Thanks much,
> Matt
>
>
> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
>> Hi,
>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>> the example at https://github.com/jbaldassari/Avro-RPC. By the way,
>> thanks for creating this example James.
>>
>> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
>> to get the server to process the requests in parallel, which is something
>> i'll need in my application. I use the DelayInjectingBidder with delay of
>> 90ms so i can watch what's happening on the server. By debugging the
>> client, I see that all requests do in fact make it to the server without
>> blocking, but watching the server log i see that each request is processed
>> sequentially. So it takes ~9s to process 100 messages.
>>
>> Digging into the NettyServer constructor, I see the default
>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>> executors. I tried overriding the worker with
>> Executors.newFixedThreadPool(8) but didn't notice any difference. Looking
>> at the threads running in Eclipse Debug mode, i see the following threads
>> which confirms that there is only one worker:
>>
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
>> Thread [main] (Running)
>> Thread [ReaderThread] (Running)
>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,
>> /0:0:0:0:0:0:0:0:45633])] (Running)
>> Thread [Avro NettyTransceiver Boss 1] (Running)
>> Thread [New I/O server worker #2-1] (Running)
>> Thread [New I/O client worker #1-1] (Running)
>>
>> I also see that the NettyTransceiver has configurable thread pools, but
>> I'm not clear on what the Transceiver's role is (client side thing?), and
>> fiddling with its ChannelFactory is causing exceptions.
>>
>> Is is possible that the behavior of the BidderTest client doesn't trigger
>> the multi-threaded server even though it's configured correctly, or that
>> the server processes all requests from a single client sequentially?
>>
>> Sidenote: i also tried using avro version 1.7.3 with the same results,
>> although the above thread pools had different names.
>>
>> I wonder if someone could shed some light on what the difference is
>> between the Server and Transceiver thread pools and how to configure them
>> for multi-threaded server processing.
>>
>> Thanks,
>> Matt
>>
>>
>
Re: avro-rpc: running multiple threads on the netty server
Posted by Matt Corgan <mc...@hotpads.com>.
hmm - i haven't heard any suggestions in the past few days... maybe a
really short version of the question: Can an Avro-rpc Netty server process
requests in parallel? Will it still process in parallel if they're from
the same client?
Thanks much,
Matt
On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <mc...@hotpads.com> wrote:
> Hi,
> I'm trying to familiarize myself with avro-rpc by running and tweaking the
> example at https://github.com/jbaldassari/Avro-RPC. By the way, thanks
> for creating this example James.
>
> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
> to get the server to process the requests in parallel, which is something
> i'll need in my application. I use the DelayInjectingBidder with delay of
> 90ms so i can watch what's happening on the server. By debugging the
> client, I see that all requests do in fact make it to the server without
> blocking, but watching the server log i see that each request is processed
> sequentially. So it takes ~9s to process 100 messages.
>
> Digging into the NettyServer constructor, I see the default ChannelFactory
> uses Executors.newCachedThreadPool() for the Boss and Worker executors. I
> tried overriding the worker with Executors.newFixedThreadPool(8) but didn't
> notice any difference. Looking at the threads running in Eclipse Debug
> mode, i see the following threads which confirms that there is only one
> worker:
>
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
> Thread [main] (Running)
> Thread [ReaderThread] (Running)
> Thread [New I/O server boss #2 ([id: 0x6a5dd151, /0:0:0:0:0:0:0:0:45633])]
> (Running)
> Thread [Avro NettyTransceiver Boss 1] (Running)
> Thread [New I/O server worker #2-1] (Running)
> Thread [New I/O client worker #1-1] (Running)
>
> I also see that the NettyTransceiver has configurable thread pools, but
> I'm not clear on what the Transceiver's role is (client side thing?), and
> fiddling with its ChannelFactory is causing exceptions.
>
> Is is possible that the behavior of the BidderTest client doesn't trigger
> the multi-threaded server even though it's configured correctly, or that
> the server processes all requests from a single client sequentially?
>
> Sidenote: i also tried using avro version 1.7.3 with the same results,
> although the above thread pools had different names.
>
> I wonder if someone could shed some light on what the difference is
> between the Server and Transceiver thread pools and how to configure them
> for multi-threaded server processing.
>
> Thanks,
> Matt
>
>