You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Tenghuan He <te...@gmail.com> on 2016/04/06 20:01:35 UTC
JMeter load test Thrift using multi threads got TTransportException:
Cannot write to null outputStream
Hi all,
I am using JMeter to load test my Thrift server.
TThreadPoolServer was used
In JMeter when I set the number of threads to more than 2000 (where each
thread will create an client with its own TSocket and TProtocol) and the
ramp up time in 1 second. After a while the client got the following error
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStre
am
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTranspo
rt.java:142)
at
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.j
ava:178)
at
org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryP
rotocol.java:106)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at
com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.send_getDataPoint(R
PCQueryProcessor.java:130)
at
com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.getDataPoint(RPCQue
ryProcessor.java:123)
at
com.corp.tsdb.common.rpc.RPCQueryProcessorClient.getDataPoint(RPCQuer
yProcessorClient.java:62)
at
com.corp.tsdb.common.rpc.loadtest.ThriftSampler.runTest(ThriftSampler
.java:45)
at
org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSample
r.java:191)
at
org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.j
ava:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Thread.java:745)
Could anyone help me?
Thanks
Tenghuan
Re: JMeter load test Thrift using multi threads got
TTransportException: Cannot write to null outputStream
Posted by Tenghuan He <te...@gmail.com>.
Hi Randy,
The Windows version used is Windows 7 ultimate edition service pack 1, and
the link you provide is a good information for me!
Yes, I run the load test on desktop just for a test and will deploy the
server on Linux Systems.
Thanks very much
On Thu, Apr 7, 2016 at 8:50 PM, Randy Abernethy <ra...@gmail.com>
wrote:
> Hey,
>
> So if the same machine can do 3,000 Java threads/sockets on one OS and not
> on another we are almost definitely running into system based limits. The
> stack trace makes me believe that you are running out of sockets (allowed
> or resources required to create). What edition of Windows 7 are you
> running? In general desktop OSes use limits designed to protect an end user
> rather than limit designed to support a server needing many client
> connections.
>
> Here is a good discussion of the Windows settings you might need to look at
> tweaking:
> http://smallvoid.com/article/winnt-tcpip-max-limit.html
>
> Or you could just try running the load test on a modern server OS (probably
> more like what you will deploy the server on anyway, yes?).
>
> Good luck!
> Randy
>
>
> On Wed, Apr 6, 2016 at 7:33 PM, Tenghuan He <te...@gmail.com> wrote:
>
> > Hi Randy
> >
> > Thanks for you replay
> >
> > 1. The environment is Windows 7 64 bit 4G memory 4 cores
> > 2. The server is running on the same machine with JMeter,
> > 3. I used this command on cygwin on my windows , the result is '256',
> does
> > that make sense?
> > 4. Both the client and server are written in Java
> >
> > I test using the same code and .jmx files in VMWare Ubuntu virtual
> machine
> > with 2G memory, raise the threads to 3000 ramp up in 1 seconds and it
> works
> > well
> > when to 5000 threads, I got three kind of errors:
> > org.apache.thrift.transport.TTransportException:
> java.net.SocketException:
> > Connection reset
> > org.apache.thrift.transport.TTransportException:
> java.net.ConnectException:
> > Connection timed out
> > org.apache.thrift.transport.TTransportException: Cannot write to null
> > outputStream
> >
> >
> >
> > On Thu, Apr 7, 2016 at 6:15 AM, Randy Abernethy <ra...@apache.org> wrote:
> >
> > > Hello,
> > >
> > > Sounds like your system is running out of resources due to shortage or
> > > limits.
> > > You could be out of memory (virt mode or phys), out of file
> descriptors,
> > > etc.
> > > Some questions:
> > >
> > > - How is the machine you are using JMeter on sized (mem/swap in
> > > particular)?
> > > - Where is the server running (if its on the machine you are running
> > > JMeter on
> > > two sockets are being used for every client thread)?
> > > - What does "ulimit -n" tell you (the max file descriptors per proc on
> > > most
> > > linux systems is 1024)?
> > > - I can see the client is Java, what is the server written in?
> > >
> > > -Randy
> > >
> > >
> > > On Wed, Apr 6, 2016 at 11:01 AM, Tenghuan He <te...@gmail.com>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am using JMeter to load test my Thrift server.
> > > > TThreadPoolServer was used
> > > >
> > > > In JMeter when I set the number of threads to more than 2000 (where
> > each
> > > > thread will create an client with its own TSocket and TProtocol) and
> > the
> > > > ramp up time in 1 second. After a while the client got the following
> > > error
> > > >
> > > > org.apache.thrift.transport.TTransportException: Cannot write to null
> > > > outputStre
> > > > am
> > > > at
> > > > org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTranspo
> > > > rt.java:142)
> > > > at
> > > > org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.j
> > > > ava:178)
> > > > at
> > > > org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryP
> > > > rotocol.java:106)
> > > > at
> > > > org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
> > > > at
> > > > org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> > > > at
> > > > com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.send_getDataPoint(R
> > > > PCQueryProcessor.java:130)
> > > > at
> > > > com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.getDataPoint(RPCQue
> > > > ryProcessor.java:123)
> > > > at
> > > > com.corp.tsdb.common.rpc.RPCQueryProcessorClient.getDataPoint(RPCQuer
> > > > yProcessorClient.java:62)
> > > > at
> > > > com.corp.tsdb.common.rpc.loadtest.ThriftSampler.runTest(ThriftSampler
> > > > .java:45)
> > > > at
> > > > org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSample
> > > > r.java:191)
> > > > at
> > > > org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.j
> > > > ava:434)
> > > > at
> > > > org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > > > Could anyone help me?
> > > > Thanks
> > > >
> > > > Tenghuan
> > > >
> > >
> >
>
Re: JMeter load test Thrift using multi threads got
TTransportException: Cannot write to null outputStream
Posted by Randy Abernethy <ra...@gmail.com>.
Hey,
So if the same machine can do 3,000 Java threads/sockets on one OS and not
on another we are almost definitely running into system based limits. The
stack trace makes me believe that you are running out of sockets (allowed
or resources required to create). What edition of Windows 7 are you
running? In general desktop OSes use limits designed to protect an end user
rather than limit designed to support a server needing many client
connections.
Here is a good discussion of the Windows settings you might need to look at
tweaking:
http://smallvoid.com/article/winnt-tcpip-max-limit.html
Or you could just try running the load test on a modern server OS (probably
more like what you will deploy the server on anyway, yes?).
Good luck!
Randy
On Wed, Apr 6, 2016 at 7:33 PM, Tenghuan He <te...@gmail.com> wrote:
> Hi Randy
>
> Thanks for you replay
>
> 1. The environment is Windows 7 64 bit 4G memory 4 cores
> 2. The server is running on the same machine with JMeter,
> 3. I used this command on cygwin on my windows , the result is '256', does
> that make sense?
> 4. Both the client and server are written in Java
>
> I test using the same code and .jmx files in VMWare Ubuntu virtual machine
> with 2G memory, raise the threads to 3000 ramp up in 1 seconds and it works
> well
> when to 5000 threads, I got three kind of errors:
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Connection reset
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection timed out
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
>
>
>
> On Thu, Apr 7, 2016 at 6:15 AM, Randy Abernethy <ra...@apache.org> wrote:
>
> > Hello,
> >
> > Sounds like your system is running out of resources due to shortage or
> > limits.
> > You could be out of memory (virt mode or phys), out of file descriptors,
> > etc.
> > Some questions:
> >
> > - How is the machine you are using JMeter on sized (mem/swap in
> > particular)?
> > - Where is the server running (if its on the machine you are running
> > JMeter on
> > two sockets are being used for every client thread)?
> > - What does "ulimit -n" tell you (the max file descriptors per proc on
> > most
> > linux systems is 1024)?
> > - I can see the client is Java, what is the server written in?
> >
> > -Randy
> >
> >
> > On Wed, Apr 6, 2016 at 11:01 AM, Tenghuan He <te...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > I am using JMeter to load test my Thrift server.
> > > TThreadPoolServer was used
> > >
> > > In JMeter when I set the number of threads to more than 2000 (where
> each
> > > thread will create an client with its own TSocket and TProtocol) and
> the
> > > ramp up time in 1 second. After a while the client got the following
> > error
> > >
> > > org.apache.thrift.transport.TTransportException: Cannot write to null
> > > outputStre
> > > am
> > > at
> > > org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTranspo
> > > rt.java:142)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.j
> > > ava:178)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryP
> > > rotocol.java:106)
> > > at
> > > org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
> > > at
> > > org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> > > at
> > > com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.send_getDataPoint(R
> > > PCQueryProcessor.java:130)
> > > at
> > > com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.getDataPoint(RPCQue
> > > ryProcessor.java:123)
> > > at
> > > com.corp.tsdb.common.rpc.RPCQueryProcessorClient.getDataPoint(RPCQuer
> > > yProcessorClient.java:62)
> > > at
> > > com.corp.tsdb.common.rpc.loadtest.ThriftSampler.runTest(ThriftSampler
> > > .java:45)
> > > at
> > > org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSample
> > > r.java:191)
> > > at
> > > org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.j
> > > ava:434)
> > > at
> > > org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Could anyone help me?
> > > Thanks
> > >
> > > Tenghuan
> > >
> >
>
Re: JMeter load test Thrift using multi threads got
TTransportException: Cannot write to null outputStream
Posted by Tenghuan He <te...@gmail.com>.
Hi Randy
Thanks for you replay
1. The environment is Windows 7 64 bit 4G memory 4 cores
2. The server is running on the same machine with JMeter,
3. I used this command on cygwin on my windows , the result is '256', does
that make sense?
4. Both the client and server are written in Java
I test using the same code and .jmx files in VMWare Ubuntu virtual machine
with 2G memory, raise the threads to 3000 ramp up in 1 seconds and it works
well
when to 5000 threads, I got three kind of errors:
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Connection reset
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection timed out
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStream
On Thu, Apr 7, 2016 at 6:15 AM, Randy Abernethy <ra...@apache.org> wrote:
> Hello,
>
> Sounds like your system is running out of resources due to shortage or
> limits.
> You could be out of memory (virt mode or phys), out of file descriptors,
> etc.
> Some questions:
>
> - How is the machine you are using JMeter on sized (mem/swap in
> particular)?
> - Where is the server running (if its on the machine you are running
> JMeter on
> two sockets are being used for every client thread)?
> - What does "ulimit -n" tell you (the max file descriptors per proc on
> most
> linux systems is 1024)?
> - I can see the client is Java, what is the server written in?
>
> -Randy
>
>
> On Wed, Apr 6, 2016 at 11:01 AM, Tenghuan He <te...@gmail.com> wrote:
>
> > Hi all,
> >
> > I am using JMeter to load test my Thrift server.
> > TThreadPoolServer was used
> >
> > In JMeter when I set the number of threads to more than 2000 (where each
> > thread will create an client with its own TSocket and TProtocol) and the
> > ramp up time in 1 second. After a while the client got the following
> error
> >
> > org.apache.thrift.transport.TTransportException: Cannot write to null
> > outputStre
> > am
> > at
> > org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTranspo
> > rt.java:142)
> > at
> > org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.j
> > ava:178)
> > at
> > org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryP
> > rotocol.java:106)
> > at
> > org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
> > at
> > org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> > at
> > com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.send_getDataPoint(R
> > PCQueryProcessor.java:130)
> > at
> > com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.getDataPoint(RPCQue
> > ryProcessor.java:123)
> > at
> > com.corp.tsdb.common.rpc.RPCQueryProcessorClient.getDataPoint(RPCQuer
> > yProcessorClient.java:62)
> > at
> > com.corp.tsdb.common.rpc.loadtest.ThriftSampler.runTest(ThriftSampler
> > .java:45)
> > at
> > org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSample
> > r.java:191)
> > at
> > org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.j
> > ava:434)
> > at
> > org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Could anyone help me?
> > Thanks
> >
> > Tenghuan
> >
>
Re: JMeter load test Thrift using multi threads got
TTransportException: Cannot write to null outputStream
Posted by Randy Abernethy <ra...@apache.org>.
Hello,
Sounds like your system is running out of resources due to shortage or
limits.
You could be out of memory (virt mode or phys), out of file descriptors,
etc.
Some questions:
- How is the machine you are using JMeter on sized (mem/swap in
particular)?
- Where is the server running (if its on the machine you are running
JMeter on
two sockets are being used for every client thread)?
- What does "ulimit -n" tell you (the max file descriptors per proc on
most
linux systems is 1024)?
- I can see the client is Java, what is the server written in?
-Randy
On Wed, Apr 6, 2016 at 11:01 AM, Tenghuan He <te...@gmail.com> wrote:
> Hi all,
>
> I am using JMeter to load test my Thrift server.
> TThreadPoolServer was used
>
> In JMeter when I set the number of threads to more than 2000 (where each
> thread will create an client with its own TSocket and TProtocol) and the
> ramp up time in 1 second. After a while the client got the following error
>
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStre
> am
> at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTranspo
> rt.java:142)
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.j
> ava:178)
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryP
> rotocol.java:106)
> at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
> at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> at
> com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.send_getDataPoint(R
> PCQueryProcessor.java:130)
> at
> com.corp.tsdb.common.rpc.RPCQueryProcessor$Client.getDataPoint(RPCQue
> ryProcessor.java:123)
> at
> com.corp.tsdb.common.rpc.RPCQueryProcessorClient.getDataPoint(RPCQuer
> yProcessorClient.java:62)
> at
> com.corp.tsdb.common.rpc.loadtest.ThriftSampler.runTest(ThriftSampler
> .java:45)
> at
> org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSample
> r.java:191)
> at
> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.j
> ava:434)
> at
> org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
> at java.lang.Thread.run(Thread.java:745)
>
> Could anyone help me?
> Thanks
>
> Tenghuan
>