You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yongle Zhang (Jira)" <ji...@apache.org> on 2020/11/12 04:34:00 UTC
[jira] [Created] (CASSANDRA-16269) Stress testing a mixed cluster
with C* 1.2.0 (seed) and 1.1.0 fails with NumberFormatException
Yongle Zhang created CASSANDRA-16269:
----------------------------------------
Summary: Stress testing a mixed cluster with C* 1.2.0 (seed) and 1.1.0 fails with NumberFormatException
Key: CASSANDRA-16269
URL: https://issues.apache.org/jira/browse/CASSANDRA-16269
Project: Cassandra
Issue Type: Bug
Reporter: Yongle Zhang
Steps to reproduce:
# Set up a mixed 1.1.0 & 1.2.0 cluster with 2 nodes (1 seed, 1.2.0 node being the seed).
# Run the stress testing tool, e.g., `/cassandra/tools/bin/stress -d 250.16.238.1,250.16.238.2 -r`, wait for finish.
Error log:
{code:java}
DEBUG [WRITE-/250.16.238.1] 2020-06-18 01:55:09,606 OutboundTcpConnection.java (line 229) attempting to connect to /250.16.238.1
DEBUG [InternalResponseStage:1] 2020-06-18 01:55:09,613 ResponseVerbHandler.java (line 44) Processing response on a callback from 122@/250.16.238.1
ERROR [InternalResponseStage:1] 2020-06-18 01:55:09,615 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[InternalResponseStage:1,5,main]
java.lang.NumberFormatException: For input string: "'115"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.math.BigInteger.<init>(BigInteger.java:470)
at java.math.BigInteger.<init>(BigInteger.java:606)
at org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:136)
at org.apache.cassandra.dht.BootStrapper$BootstrapTokenCallback.response(BootStrapper.java:213)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
DEBUG [InternalResponseStage:2] 2020-06-18 01:55:39,610 ResponseVerbHandler.java (line 44) Processing response on a callback from 262@/250.16.238.1
ERROR [InternalResponseStage:2] 2020-06-18 01:55:39,611 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[InternalResponseStage:2,5,main]
java.lang.NumberFormatException: For input string: "^@'115"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.math.BigInteger.<init>(BigInteger.java:470)
at java.math.BigInteger.<init>(BigInteger.java:606)
at org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:136)
at org.apache.cassandra.dht.BootStrapper$BootstrapTokenCallback.response(BootStrapper.java:213)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)ERROR [main] 2020-06-18 01:57:39,614 AbstractCassandraDaemon.java (line 370) Exception encountered during startup
java.lang.RuntimeException: Bootstrap failed, could not obtain token from: /250.16.238.1
at org.apache.cassandra.dht.BootStrapper.getBootstrapTokenFrom(BootStrapper.java:177)
at org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:110)
at org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:104)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:599)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:515)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:407)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:231)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
ERROR [main] 2020-06-18 01:58:13,206 AbstractCassandraDaemon.java (line 370) Exception encountered during startup
java.lang.RuntimeException: No other nodes seen! Unable to bootstrap.If you intended to start a single-node cluster, you should make sure your broadcast_address (or listen_address) is listed as a seed. Otherwise, you need to determine why the seed being contacted has no knowledge of the rest of the cluster. Usually, this can be solved by giving all nodes the same seed list.
at org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.java:127)
at org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:109)
at org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:104)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:599)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:515)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:407)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:231)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
INFO [StorageServiceShutdownHook] 2020-06-18 01:58:13,213 MessagingService.java (line 539) Waiting for messaging service to quiesce
INFO [ACCEPT-/250.16.238.3] 2020-06-18 01:58:13,215 MessagingService.java (line 695) MessagingService shutting down server thread.
DEBUG [GossipTasks:1] 2020-06-18 01:58:13,244 DebuggableThreadPoolExecutor.java (line 167) Task cancelled
java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask.report(FutureTask.java:121)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:159)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:123)
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:50)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Root cause analysis:
In 1.2.0, when a `MessageOut<T>` class, where `T` is the type of its payload, is serialized, it will first write the logical header part of it (which is compatible with 1.1.0), and then write a int to indicate the length of its payload, and then serialize its payload. When `T` is `String`, it will call `DataOutput.writeUTF`, which will write a short to indicate the length of the string.
In 1.1.0, the incoming byte stream will be serialized as `Message` class, which will first serialize the header part, and then read a int as the length of the following payload, and then read the following raw bytes as its payload. The problem is that the payload will then be directly encoded as a UTF string, so *the short will be considered as part of the string*. For example, this log shows:
{code:java}
WARN [InternalResponseStage:3] 2020-07-10 08:14:08,903 RandomPartitioner.java (line 145) Token.TokenFactory<BigInteger>.fromString: string = �'163345392646151285100494393711260197750, bytes = 0 39 49 54 51 51 52 53 51 57 50 54 52 54 49 53 49 50 56 53 49 48 48 52 57 52 51 57 51 55 49 49 50 54 48 49 57 55 55 53 48
{code}
The first two byte 0 39 is the big endian short representation of number 39, which is exactly the length of the following string. These two bytes can’t be parsed as a number, so that a NumberFormatException happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org