You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kazuo YAGI <ky...@zynga.co.jp> on 2011/04/04 11:14:20 UTC
Re: Error messages after rolling updating cassandra from 0.7.0 to
0.7.2
>> Solution: upgrade to 0.7.4, run scrub
Although I upgraded all my cassandra nodes from 0.7.0 to 0.7.4 and
ran nodetool scrub to all keyspaces, this EOFException error messages
didn't go away. Do you have any ideas how to deal with it next?
Besides, it would be really useful if I could know whether or not
these error messages are ignorable, because our application has been
working well before and after upgrading.
Thanks,
Kazuo
(11/03/28 11:19), Kazuo YAGI wrote:
> I really thank you for your information.
> I'll try to upgrade 0.7.4 and run scrub.
>
> (11/03/25 22:39), Jonathan Ellis wrote:
>> from NEWS.txt:
>>
>> Upgrading
>> ---------
>> - 0.7.1 and 0.7.2 shipped with a bug that caused incorrect row-level
>> bloom filters to be generated when compacting sstables generated
>> with earlier versions. This would manifest in IOExceptions during
>> column name-based queries. 0.7.3 provides "nodetool scrub" to
>> rebuild sstables with correct bloom filters, with no data lost.
>> (If your cluster was never on 0.7.0 or earlier, you don't have to
>> worry about this.) Note that nodetool scrub will snapshot your
>> data files before rebuilding, just in case.
>>
>> Solution: upgrade to 0.7.4, run scrub
>>
>> 2011/3/25 Kazuo YAGI<ky...@zynga.co.jp>:
>>> Hi everyone,
>>>
>>> I'm running 10 cassandra nodes in total and updated 5 nodes
>>> from 0.7.0 to 0.7.2 one by one like below.
>>>
>>> # sudo /etc/init.d/cassandra stop
>>> # sudo yum update cassandra-0.7.2-4
>>> # sudo /etc/init.d/cassandra start
>>>
>>> After updating, those 5 updated nodes keep outputting error messages to
>>> /var/lib/cassandra/log/system.log every exactly 5 minutes.
>>>
>>> Do you have any idea about this error message? Thanks for your support.
>>>
>>> /var/lib/cassandra/log/system.log
>>> ------
>>> ERROR [Thread-45] 2011-03-23 21:21:29,125
>>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>>> Thread[Thread-45,5,main]
>>> java.io.IOError: java.io.EOFException
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>>
>>> Caused by: java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>>
>>> ERROR [Thread-51] 2011-03-23 21:26:29,227
>>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>>> Thread[Thread-51,5,main]
>>> java.io.IOError: java.io.EOFException
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>>
>>> Caused by: java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>>
>>> ------
>>>
>>> Best,
>>> Kazuo YAGI
>>>
>>
>>
>>
>
>
Re: Error messages after rolling updating cassandra from 0.7.0 to
0.7.2
Posted by Kazuo YAGI <ky...@zynga.co.jp>.
That's it! Nagios causes this message. I should have noticed it
when I found the message appeared every exactly 5 minutes.
define command{
command_name check_cassandra_node
command_line $USER1$/check_tcp -H $HOSTADDRESS$ -p 7000 -t 5 -E -M crit
}
The issue is settled now. Thank you for your advice.
Kazuo
(11/04/07 21:15), Kazuo YAGI wrote:
> Thank both of you. I'll look it up from another view including
> non-cassandra processes on the servers. Just guessing Nagios or
> iptables or others causes it.
>
> Kazuo
>
> (11/04/05 22:55), Sasha Dolgy wrote:
>> I've been seeing this EOF in my system.log file occasionally as well.
>> Doesn't seem to be causing harm:
>>
>> ERROR [Thread-22] 2011-04-05 20:37:22,562 AbstractCassandraDaemon.java
>> (line 112) Fatal exception in thread Thread[Thread-22,5,main]
>> java.io.IOError: java.io.EOFException
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>
>>
>> Firewall rules prevent anything but cassandra instances accessing
>> cassandra instances on port 7000 ...
>>
>> This is with 0.7.4
>>
>> -sd
>>
>> On Tue, Apr 5, 2011 at 3:47 PM, Jonathan Ellis<jb...@gmail.com> wrote:
>>> Oops, I saw "EOFException" and jumped to "scrub."
>>>
>>> But your EOF is coming from TCP. Something (almost certainly a
>>> non-cassandra process) is connecting to the internal Cassandra
>>> communication port (the one that defaults to 7000) and disconnecting.
>>>
>>> On Mon, Apr 4, 2011 at 4:14 AM, Kazuo YAGI<ky...@zynga.co.jp> wrote:
>>>>>> Solution: upgrade to 0.7.4, run scrub
>>>>
>>>> Although I upgraded all my cassandra nodes from 0.7.0 to 0.7.4 and
>>>> ran nodetool scrub to all keyspaces, this EOFException error messages
>>>> didn't go away. Do you have any ideas how to deal with it next?
>>>>
>>>> Besides, it would be really useful if I could know whether or not
>>>> these error messages are ignorable, because our application has been
>>>> working well before and after upgrading.
>>>>
>>>> Thanks,
>>>> Kazuo
>>
>
Re: Error messages after rolling updating cassandra from 0.7.0 to
0.7.2
Posted by Kazuo YAGI <ky...@zynga.co.jp>.
Thank both of you. I'll look it up from another view including
non-cassandra processes on the servers. Just guessing Nagios or
iptables or others causes it.
Kazuo
(11/04/05 22:55), Sasha Dolgy wrote:
> I've been seeing this EOF in my system.log file occasionally as well.
> Doesn't seem to be causing harm:
>
> ERROR [Thread-22] 2011-04-05 20:37:22,562 AbstractCassandraDaemon.java
> (line 112) Fatal exception in thread Thread[Thread-22,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>
> Firewall rules prevent anything but cassandra instances accessing
> cassandra instances on port 7000 ...
>
> This is with 0.7.4
>
> -sd
>
> On Tue, Apr 5, 2011 at 3:47 PM, Jonathan Ellis<jb...@gmail.com> wrote:
>> Oops, I saw "EOFException" and jumped to "scrub."
>>
>> But your EOF is coming from TCP. Something (almost certainly a
>> non-cassandra process) is connecting to the internal Cassandra
>> communication port (the one that defaults to 7000) and disconnecting.
>>
>> On Mon, Apr 4, 2011 at 4:14 AM, Kazuo YAGI<ky...@zynga.co.jp> wrote:
>>>>> Solution: upgrade to 0.7.4, run scrub
>>>
>>> Although I upgraded all my cassandra nodes from 0.7.0 to 0.7.4 and
>>> ran nodetool scrub to all keyspaces, this EOFException error messages
>>> didn't go away. Do you have any ideas how to deal with it next?
>>>
>>> Besides, it would be really useful if I could know whether or not
>>> these error messages are ignorable, because our application has been
>>> working well before and after upgrading.
>>>
>>> Thanks,
>>> Kazuo
>
Re: Error messages after rolling updating cassandra from 0.7.0 to 0.7.2
Posted by Sasha Dolgy <sd...@gmail.com>.
I've been seeing this EOF in my system.log file occasionally as well.
Doesn't seem to be causing harm:
ERROR [Thread-22] 2011-04-05 20:37:22,562 AbstractCassandraDaemon.java
(line 112) Fatal exception in thread Thread[Thread-22,5,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
Firewall rules prevent anything but cassandra instances accessing
cassandra instances on port 7000 ...
This is with 0.7.4
-sd
On Tue, Apr 5, 2011 at 3:47 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Oops, I saw "EOFException" and jumped to "scrub."
>
> But your EOF is coming from TCP. Something (almost certainly a
> non-cassandra process) is connecting to the internal Cassandra
> communication port (the one that defaults to 7000) and disconnecting.
>
> On Mon, Apr 4, 2011 at 4:14 AM, Kazuo YAGI <ky...@zynga.co.jp> wrote:
>>>> Solution: upgrade to 0.7.4, run scrub
>>
>> Although I upgraded all my cassandra nodes from 0.7.0 to 0.7.4 and
>> ran nodetool scrub to all keyspaces, this EOFException error messages
>> didn't go away. Do you have any ideas how to deal with it next?
>>
>> Besides, it would be really useful if I could know whether or not
>> these error messages are ignorable, because our application has been
>> working well before and after upgrading.
>>
>> Thanks,
>> Kazuo
Re: Error messages after rolling updating cassandra from 0.7.0 to 0.7.2
Posted by Jonathan Ellis <jb...@gmail.com>.
Oops, I saw "EOFException" and jumped to "scrub."
But your EOF is coming from TCP. Something (almost certainly a
non-cassandra process) is connecting to the internal Cassandra
communication port (the one that defaults to 7000) and disconnecting.
On Mon, Apr 4, 2011 at 4:14 AM, Kazuo YAGI <ky...@zynga.co.jp> wrote:
>>> Solution: upgrade to 0.7.4, run scrub
>
> Although I upgraded all my cassandra nodes from 0.7.0 to 0.7.4 and
> ran nodetool scrub to all keyspaces, this EOFException error messages
> didn't go away. Do you have any ideas how to deal with it next?
>
> Besides, it would be really useful if I could know whether or not
> these error messages are ignorable, because our application has been
> working well before and after upgrading.
>
> Thanks,
> Kazuo
>
> (11/03/28 11:19), Kazuo YAGI wrote:
>>
>> I really thank you for your information.
>> I'll try to upgrade 0.7.4 and run scrub.
>>
>> (11/03/25 22:39), Jonathan Ellis wrote:
>>>
>>> from NEWS.txt:
>>>
>>> Upgrading
>>> ---------
>>> - 0.7.1 and 0.7.2 shipped with a bug that caused incorrect row-level
>>> bloom filters to be generated when compacting sstables generated
>>> with earlier versions. This would manifest in IOExceptions during
>>> column name-based queries. 0.7.3 provides "nodetool scrub" to
>>> rebuild sstables with correct bloom filters, with no data lost.
>>> (If your cluster was never on 0.7.0 or earlier, you don't have to
>>> worry about this.) Note that nodetool scrub will snapshot your
>>> data files before rebuilding, just in case.
>>>
>>> Solution: upgrade to 0.7.4, run scrub
>>>
>>> 2011/3/25 Kazuo YAGI<ky...@zynga.co.jp>:
>>>>
>>>> Hi everyone,
>>>>
>>>> I'm running 10 cassandra nodes in total and updated 5 nodes
>>>> from 0.7.0 to 0.7.2 one by one like below.
>>>>
>>>> # sudo /etc/init.d/cassandra stop
>>>> # sudo yum update cassandra-0.7.2-4
>>>> # sudo /etc/init.d/cassandra start
>>>>
>>>> After updating, those 5 updated nodes keep outputting error messages to
>>>> /var/lib/cassandra/log/system.log every exactly 5 minutes.
>>>>
>>>> Do you have any idea about this error message? Thanks for your support.
>>>>
>>>> /var/lib/cassandra/log/system.log
>>>> ------
>>>> ERROR [Thread-45] 2011-03-23 21:21:29,125
>>>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>>>> Thread[Thread-45,5,main]
>>>> java.io.IOError: java.io.EOFException
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>>>
>>>> Caused by: java.io.EOFException
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>>>
>>>> ERROR [Thread-51] 2011-03-23 21:26:29,227
>>>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>>>> Thread[Thread-51,5,main]
>>>> java.io.IOError: java.io.EOFException
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>>>
>>>> Caused by: java.io.EOFException
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>>>
>>>> ------
>>>>
>>>> Best,
>>>> Kazuo YAGI
>>>>
>>>
>>>
>>>
>>
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com