You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by John Watson <jo...@disqus.com> on 2013/04/14 23:20:50 UTC
1.1.9 to 1.2.3 upgrade issue
Started doing a rolling upgrade of nodes from 1.1.9 to 1.2.3 and nodes on
1.1.9 started flooding this error:
Exception in thread Thread[RequestResponseStage:19496,5,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:155)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more
As I understand the Hints CF changed from 1.1.x to 1.2.x so I assume that's
the cause of the 1.2.3 nodes flooding (for various IPs still being 1.1.9):
Unable to store hint for host with missing ID, /10.37.62.71 (old node?)
Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
Thanks,
John
Re: 1.1.9 to 1.2.3 upgrade issue
Posted by John Watson <jo...@disqus.com>.
Yes, the exceptions were only on the 1.1.9 nodes.
Unfortunately, couldn't complete the upgrade because it was adversely
affecting the applications using the cluster during that time.
Best,
John
On Tue, Apr 16, 2013 at 2:03 PM, aaron morton <aa...@thelastpickle.com>wrote:
> Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
>
> Definitely supported.
>
> The error is from the coordinator processing the response from a replica.
> The size of the digest in the response has been miss-reported.
>
> Was this only reported on the 1.1.9 nodes ?
> Did you complete the upgrade ?
>
> I see you created https://issues.apache.org/jira/browse/CASSANDRA-5476 thanks.
> I cannot see any resolved issues in the 1.2 branch that would cover this.
>
> Cheers
>
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/04/2013, at 9:20 AM, John Watson <jo...@disqus.com> wrote:
>
> Started doing a rolling upgrade of nodes from 1.1.9 to 1.2.3 and nodes on
> 1.1.9 started flooding this error:
>
> Exception in thread Thread[RequestResponseStage:19496,5,main]
> java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at
> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:155)
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
> at
> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
> at
> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
> ... 6 more
>
> As I understand the Hints CF changed from 1.1.x to 1.2.x so I assume
> that's the cause of the 1.2.3 nodes flooding (for various IPs still being
> 1.1.9):
>
> Unable to store hint for host with missing ID, /10.37.62.71 (old
> node?)
>
> Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
>
> Thanks,
>
> John
>
>
>
Re: 1.1.9 to 1.2.3 upgrade issue
Posted by aaron morton <aa...@thelastpickle.com>.
> Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
Definitely supported.
The error is from the coordinator processing the response from a replica. The size of the digest in the response has been miss-reported.
Was this only reported on the 1.1.9 nodes ?
Did you complete the upgrade ?
I see you created https://issues.apache.org/jira/browse/CASSANDRA-5476 thanks. I cannot see any resolved issues in the 1.2 branch that would cover this.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 15/04/2013, at 9:20 AM, John Watson <jo...@disqus.com> wrote:
> Started doing a rolling upgrade of nodes from 1.1.9 to 1.2.3 and nodes on 1.1.9 started flooding this error:
>
> Exception in thread Thread[RequestResponseStage:19496,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:155)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
> ... 6 more
>
> As I understand the Hints CF changed from 1.1.x to 1.2.x so I assume that's the cause of the 1.2.3 nodes flooding (for various IPs still being 1.1.9):
>
> Unable to store hint for host with missing ID, /10.37.62.71 (old node?)
>
> Is this a known issue? Or rolling upgrade form 1.1.x to 1.2.x not possible?
>
> Thanks,
>
> John