Bulk loading errors with 1.0.8

[Benoit Perroud <be...@noisette.ch>]

Hi All,

I'm experiencing the following errors while bulk loading data into a cluster

ERROR [Thread-23] 2012-04-05 09:58:12,252 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-23,5,main]
java.lang.RuntimeException: Insufficient disk space to flush
7813594056494754913 bytes
        at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:635)
        at org.apache.cassandra.streaming.StreamIn.getContextMapping(StreamIn.java:92)
        at org.apache.cassandra.streaming.IncomingStreamReader.<init>(IncomingStreamReader.java:68)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)

Here I'm not really sure I was able to generate 7 exa bytes of data ;)


ERROR [Thread-46] 2012-04-05 09:58:14,453 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-46,5,main]
java.lang.NullPointerException
        at org.apache.cassandra.io.sstable.SSTable.getMinimalKey(SSTable.java:156)
        at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334)
        at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:302)
        at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:155)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:89)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)

This one sounds like a null key added to the SSTable at some point,
but I'm rather confident I'm checking for key nullity.

Errors are seen in different nodes. Some nodes succeeded. All 5
cluster nodes run 1.0.8 with JNA enabled.

Basically I'm generating SSTables in a Hadoop Reducer, storing them in
HDFS. After the job finished, I download them back into a single node
and streaming the files into Cassandra (Yes, I definitely need to try
1.1).

Do someone have a hint or a pointer where to start looking ?

Thanks,

Benoit.

Re: Bulk loading errors with 1.0.8

[Jonathan Ellis <jb...@gmail.com>]

On Thu, Apr 5, 2012 at 10:58 AM, Benoit Perroud <be...@noisette.ch> wrote:
> ERROR [Thread-23] 2012-04-05 09:58:12,252 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-23,5,main]
> java.lang.RuntimeException: Insufficient disk space to flush
> 7813594056494754913 bytes
>        at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:635)
>        at org.apache.cassandra.streaming.StreamIn.getContextMapping(StreamIn.java:92)
>        at org.apache.cassandra.streaming.IncomingStreamReader.<init>(IncomingStreamReader.java:68)
>        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
>        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
>
> Here I'm not really sure I was able to generate 7 exa bytes of data ;)

The bulk loader told the Cassandra node, "I have 7EB of data for you."
 And the C* node threw this error.  So you need to troubleshoot the
bulk loader side.

If you feel lucky, we've done some work on streaming in 1.1 to make it
more robust, but I don't recognize this specific problem so I can't
say for sure if 1.1 would help.

> ERROR [Thread-46] 2012-04-05 09:58:14,453 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-46,5,main]
> java.lang.NullPointerException
>        at org.apache.cassandra.io.sstable.SSTable.getMinimalKey(SSTable.java:156)
>        at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334)
>        at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:302)
>        at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:155)
>        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:89)
>        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
>        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
>
> This one sounds like a null key added to the SSTable at some point,
> but I'm rather confident I'm checking for key nullity.

The stacktrace indicates an error with the very first key in the
sstable, if that helps.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com