You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dikang Gu (JIRA)" <ji...@apache.org> on 2017/03/08 22:25:38 UTC
[jira] [Reopened] (CASSANDRA-8681) cassandra-stress fails after
single ReadTimeoutException
[ https://issues.apache.org/jira/browse/CASSANDRA-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dikang Gu reopened CASSANDRA-8681:
----------------------------------
Reproduced In: 3.9, 2.2.0 beta 1 (was: 2.2.0 beta 1)
I want to re-open this issue, because I constantly hit this when benchmarking 3.9 branch.
command I use:
{code}
/tools/bin/cassandra-stress user n=1000000000 profile=/root/cassandra_stress.yaml ops\(insert=2,singlepost=1\) cl=ONE -node 2401:db00:21:2121:face:0:13:0 -rate threads=100 -log file=~/cass-stress-1b.log
{code}
error:
{code}
total, 123245370, 137904, 119461, 119461, 0.7, 0.5, 1.9, 3.5, 6.6, 21.4, 980.2, 0.13322, 0, 0, 0, 0, 0, 0
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
insert, 82257602, 79373, 79373, 79373, 0.7, 0.4, 1.7, 3.3, 18.2, 133.2, 981.3, 0.13313, 0, 0, 0, 0, 0, 0
singlepost, 41114666, 40220, 24092, 24092, 1.0, 0.7, 2.1, 4.0, 129.1, 131.9, 981.3, 0.13313, 0, 0, 0, 0, 0, 0
total, 123372268, 119590, 103462, 103462, 0.8, 0.5, 1.9, 3.6, 18.7, 133.2, 981.3, 0.13313, 0, 0, 0, 0, 0, 0
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency ONE (1 responses were required but only 0 replica responded, 1 failed)
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.io.IOException: Operation x10 on key(s) [|=VO
oq3# H"gz`_U%A'qqp3uH{C(ux-1^6>9=PYq]M:+]: Error executing: (NoSuchElementException)
at org.apache.cassandra.stress.Operation.error(Operation.java:138)
at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:156)
at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:434)
FAILURE
java.lang.RuntimeException: Failed to execute stress action
at org.apache.cassandra.stress.StressAction.run(StressAction.java:86)
at org.apache.cassandra.stress.Stress.run(Stress.java:133)
at org.apache.cassandra.stress.Stress.main(Stress.java:61)
[root@cassandrarocksdbtest1-prn <lxc> /data/apache-cassandra-3.9 ] #
{code}
> cassandra-stress fails after single ReadTimeoutException
> --------------------------------------------------------
>
> Key: CASSANDRA-8681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8681
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Robert Stupp
> Assignee: T Jake Luciani
> Labels: stress
>
> After a single exception is propagated from the Java Driver, cassandra-stress prints out some {{NoSuchElementException}}s and stops working.
> {noformat}
> Running [singlepost, timeline] with 36 threads 2 minutes
> total ops , adj row/s, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, gc: #, max ms, sum ms, sdv ms, mb
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (timeout while waiting for repair of inconsistent replica)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (timeout while waiting for repair of inconsistent replica)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> java.util.NoSuchElementException
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (timeout while waiting for repair of inconsistent replica)
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.io.IOException: Operation x10 on key(s) [Z$Vq"zj\Aw}KG2>9$TY(!">e)
> m#$D1x??]D?2S]: Error executing: (NoSuchElementException)
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> java.util.NoSuchElementException
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)java.util.NoSuchElementException
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)java.util.NoSuchElementException
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [ot^XJz*C{
> HYUg+"' j^36`
> KX aj0n;]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [\!d,*&M D4 *^%=c MPZBZ+%W=2J
> Ic~s1/[Ba?3}V]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> f\llVm8IMYExceptX"RCSD}:Hx~0-n x&Q on key(s) [3+`&BP0yu,sdQ!s+CYw]ILLPk v)^IJ<z`>SU-
> `]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [KDIWY$$D?.+(y2iT="toY
> 5\.^
> _dV]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> [(>V 1]: Error executing: (NoSuchElementException)4f}n-K|.ly6q&5-/QLgu~2V{fmw5OJ;+!(v' <T?#A$
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [$:YcLz5^?(v$aQR7b~JEjd,|]X8R><6u@P(zCTS"d0Z]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [6-1YKka#HB2^o.g{9Mvq({ I9-I^!5S?jhNA>h#A434]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [/aW4D-aS
> lv2)fa_0I+4$
> C&]}WedM^B9D<v.\]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> java.io.IOException: Operation x10 on key(s) [lk/!
> py%
> u"@$%wO4)2q]#j+1kE+0
> x.8lvA}"R6}f_]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> $ e}*#D*T&*j Ction: Operation x10 on key(s) [
> T6{DlsMm' u.z]: Error executing: (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:211)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
> at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)
> at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)