You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Suhail Doshi <su...@mixpanel.com> on 2010/02/01 18:31:38 UTC
Re: Frequent timeout exceptions on Cassandra
So it turns out it's not socket related and it seems to only happen on my
second new node only not the first one with a higher load.
I decommissioned the new node and replaced it with another new node from
scratch but the issue persists: clients continue to timeout on it and
eventual the node becomes unavailable after a while requiring a restart.
I've gone as far as I can trying to debug it so would greatly appreciate any
insight. Let me know if there's anything else you need to know.
Suhail
On Sun, Jan 31, 2010 at 2:55 PM, Suhail Doshi <su...@mixpanel.com> wrote:
> Encountering some new issues while running clients against Cassandra. For some reason as time goes Cassandra in my case starts to throw timeout exceptions to the point where even if one client is connected it processes things *very* slow and often throws the timeout exception frequently. My clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
>
> Does anyone have a clue what might cause the timeout errors? Things seem to get progressively worse as time goes on.
>
> I wrote my own client using Thrift's python output of the thrift interface. Pretty standard and close to the example. open transport -> execute query -> close transport. I am wondering if possibly I shouldn't close transport on every query and leave it open. Potentially opening and closing the socket to the thrift server could cause problems at scale?
>
> I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of memory on the 2 64-bit 8 G machines.
>
>
>
> First run
>
> ----------------------
> total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
> Second run (ho later)
>
> ----------------------
> total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process Inserter-22:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
> Third run (3 hours later)Note: According to jconsole write operations were < 10,000 but during the first run I saw them peak at 80,000. Huge difference.----------------------
> total,interval_op_rate,avg_latency,elapsed_timeProcess Inserter-26:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-4:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 600, in recv_batch_insert
> raise result.ueUnavailableException: UnavailableException()Process Inserter-6:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 600, in recv_batch_insert
> raise result.ueUnavailableException: UnavailableException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 600, in recv_batch_insert
> raise result.ueUnavailableException: UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process Inserter-9:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process Inserter-45:Traceback (most recent call last):Process Inserter-23:Traceback (most recent call last):Process Inserter-29:Process Inserter-32:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-20:Traceback (most recent call last):Process Inserter-42:Process Inserter-33:Traceback (most recent call last):Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTraceback (most recent call last):TimedOutException: TimedOutException()
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-34:Traceback (most recent call last):Process Inserter-8:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-24:Traceback (most recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call last):
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-36:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-1:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call last):
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> raise result.teTimedOutException: TimedOutException()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-37:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insertProcess Inserter-38:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> raise result.teTimedOutException: TimedOutException()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-7:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process Inserter-18:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-39:Traceback (most recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-12:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-19:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-35:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-41:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-15:
> self.run()Traceback (most recent call last):
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
>
>
> 44828,2,5.20553808377,22044976,14,2.0302237675,230Process Inserter-31:Traceback (most recent call last):Process Inserter-11:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-30:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call last):
> self.run()
> File "stress.py", line 167, in run
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-46:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()45324,34,0.361710271616,240Process Inserter-44:Traceback (most recent call last):Process Inserter-21:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-22:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-49:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()45394,7,2.14389359951,250Process Inserter-10:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process Inserter-13:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process Inserter-5:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process Inserter-28:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-3:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-16:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.te
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insertTimedOutException: TimedOutException()
> raise result.teTimedOutException: TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process Inserter-25:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-17:Traceback (most recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process Inserter-50:Process Inserter-40:Traceback (most recent call last):Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-14:
> self.run()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "stress.py", line 167, in run
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertTraceback (most recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()67902,11,0.370109583424,560Process Inserter-48:Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
> self.recv_batch_insert()
> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
>
> Extra information:
>
> 1st no clients were connected when it was run
> 2nd run, stress test was conducted when there were 50+ clients doing 7k read, 3-4k write operations
> 3rd was run after clients were killed as problems showed up
> After 3rd run looked at tpstats for seed-node:Pool Name Active Pending CompletedFILEUTILS-DELETE-POOL 0 0 442STREAM-STAGE 0 0 0RESPONSE-STAGE 0 0 1427177ROW-READ-STAGE 0 0 3705481LB-OPERATIONS 0 0 0COMMITLOG 1 0 5133614MESSAGE-DESERIALIZER-POOL 0 0 3456220GMFD 0 0 34882LB-TARGET 0 0 0CONSISTENCY-MANAGER 0 0 0ROW-MUTATION-STAGE 0 0 5452268MESSAGE-STREAMING-POOL 0 0 0LOAD-BALANCER-STAGE 0 0 0FLUSH-SORTER-POOL 0 0 118MEMTABLE-POST-FLUSHER 0 0 118FLUSH-WRITER-POOL 0 0 118AE-SERVICE-STAGE 0 0 1HINTED-HANDOFF-POOL 0 0 41
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Frequent timeout exceptions on Cassandra
Posted by Suhail Doshi <su...@mixpanel.com>.
One last thing, timeouts start happening to happen more frequently either
with time or just more reads/writes.
Suhail
On Mon, Feb 1, 2010 at 9:31 AM, Suhail Doshi <su...@mixpanel.com> wrote:
> So it turns out it's not socket related and it seems to only happen on my
> second new node only not the first one with a higher load.
>
> I decommissioned the new node and replaced it with another new node from
> scratch but the issue persists: clients continue to timeout on it and
> eventual the node becomes unavailable after a while requiring a restart.
>
> I've gone as far as I can trying to debug it so would greatly appreciate
> any insight. Let me know if there's anything else you need to know.
>
> Suhail
>
> On Sun, Jan 31, 2010 at 2:55 PM, Suhail Doshi <su...@mixpanel.com>wrote:
>
>> Encountering some new issues while running clients against Cassandra. For some reason as time goes Cassandra in my case starts to throw timeout exceptions to the point where even if one client is connected it processes things *very* slow and often throws the timeout exception frequently. My clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
>>
>> Does anyone have a clue what might cause the timeout errors? Things seem to get progressively worse as time goes on.
>>
>> I wrote my own client using Thrift's python output of the thrift interface. Pretty standard and close to the example. open transport -> execute query -> close transport. I am wondering if possibly I shouldn't close transport on every query and leave it open. Potentially opening and closing the socket to the thrift server could cause problems at scale?
>>
>> I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of memory on the 2 64-bit 8 G machines.
>>
>>
>>
>>
>> First run
>>
>> ----------------------
>> total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
>> Second run (ho later)
>>
>> ----------------------
>> total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process Inserter-22:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
>> Third run (3 hours later)Note: According to jconsole write operations were < 10,000 but during the first run I saw them peak at 80,000. Huge difference.----------------------
>> total,interval_op_rate,avg_latency,elapsed_timeProcess Inserter-26:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-4:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 600, in recv_batch_insert
>> raise result.ueUnavailableException: UnavailableException()Process Inserter-6:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 600, in recv_batch_insert
>> raise result.ueUnavailableException: UnavailableException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 600, in recv_batch_insert
>> raise result.ueUnavailableException: UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process Inserter-9:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process Inserter-45:Traceback (most recent call last):Process Inserter-23:Traceback (most recent call last):Process Inserter-29:Process Inserter-32:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-20:Traceback (most recent call last):Process Inserter-42:Process Inserter-33:Traceback (most recent call last):Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTraceback (most recent call last):TimedOutException: TimedOutException()
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-34:Traceback (most recent call last):Process Inserter-8:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-24:Traceback (most recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call last):
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-36:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-1:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call last):
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> raise result.teTimedOutException: TimedOutException()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-37:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insertProcess Inserter-38:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> raise result.teTimedOutException: TimedOutException()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-7:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process Inserter-18:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-39:Traceback (most recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-12:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-19:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-35:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-41:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-15:
>> self.run()Traceback (most recent call last):
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
>>
>>
>> 44828,2,5.20553808377,22044976,14,2.0302237675,230Process Inserter-31:Traceback (most recent call last):Process Inserter-11:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-30:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call last):
>> self.run()
>> File "stress.py", line 167, in run
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-46:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()45324,34,0.361710271616,240Process Inserter-44:Traceback (most recent call last):Process Inserter-21:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-22:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-49:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()45394,7,2.14389359951,250Process Inserter-10:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process Inserter-13:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process Inserter-5:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process Inserter-28:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-3:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertProcess Inserter-16:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.te
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insertTimedOutException: TimedOutException()
>> raise result.teTimedOutException: TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process Inserter-25:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-17:Traceback (most recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process Inserter-50:Process Inserter-40:Traceback (most recent call last):Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-14:
>> self.run()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "stress.py", line 167, in run
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insertTraceback (most recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()67902,11,0.370109583424,560Process Inserter-48:Traceback (most recent call last):
>> File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py", line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 574, in batch_insert
>> self.recv_batch_insert()
>> File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
>>
>> Extra information:
>>
>> 1st no clients were connected when it was run
>> 2nd run, stress test was conducted when there were 50+ clients doing 7k read, 3-4k write operations
>> 3rd was run after clients were killed as problems showed up
>> After 3rd run looked at tpstats for seed-node:Pool Name Active Pending CompletedFILEUTILS-DELETE-POOL 0 0 442STREAM-STAGE 0 0 0RESPONSE-STAGE 0 0 1427177ROW-READ-STAGE 0 0 3705481LB-OPERATIONS 0 0 0COMMITLOG 1 0 5133614MESSAGE-DESERIALIZER-POOL 0 0 3456220GMFD 0 0 34882LB-TARGET 0 0 0CONSISTENCY-MANAGER 0 0 0ROW-MUTATION-STAGE 0 0 5452268MESSAGE-STREAMING-POOL 0 0 0LOAD-BALANCER-STAGE 0 0 0FLUSH-SORTER-POOL 0 0 118MEMTABLE-POST-FLUSHER 0 0 118FLUSH-WRITER-POOL 0 0 118AE-SERVICE-STAGE 0 0 1HINTED-HANDOFF-POOL 0 0 41
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com