You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Martin Bligh (JIRA)" <ji...@apache.org> on 2014/04/10 15:23:15 UTC
[jira] [Comment Edited] (CASSANDRA-6981) java.io.EOFException from
Cassandra when doing select
[ https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965328#comment-13965328 ]
Martin Bligh edited comment on CASSANDRA-6981 at 4/10/14 1:23 PM:
------------------------------------------------------------------
So this is a little tricky, because it's proprietary data and I've changed things around a bit since then. Basically what I was doing was on a desktop machine with 32GB of RAM and just one disk (regular HDD, not SSD), created about 16 tables, all the same, each with about 5 text fields and 5 binary fields. Most of those fields had a secondary index. Then insert into all the tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall over like this ....
PS. I've never seen this again since disabling mmap access, but am fairly sure that's great for performance.
was (Author: mbligh):
So this is a little tricky, because it's proprietary data and I've changed things around a bit since then. Basically what I was doing was on a desktop machine with 32GB of RAM and just one disk (regular HDD, not SSD), created about 16 tables, all the same, each with about 5 text fields and 5 binary fields. Most of those fields had a secondary index. Then insert into all the tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall over like this ....
> java.io.EOFException from Cassandra when doing select
> -----------------------------------------------------
>
> Key: CASSANDRA-6981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6981
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16
> Reporter: Martin Bligh
>
> Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16
> I have a cassandra keyspace with about 12 tables that are all the same.
> If I load 100,000 rows or so into a couple of those tables in Cassandra, it works fine.
> If I load a larger dataset, after a while one of the tables won't do lookups any more (not always the same one).
> {noformat}
> SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221';
> {noformat}
> results in "Request did not complete within rpc_timeout."
> where "mid" is the primary key (varchar). If I look at the logs, it has an EOFException ... presumably it's running out of some resource (it's definitely not out of disk space)
> Sometimes it does this on secondary indexes too: dropping and rebuilding the index will fix it for a while. When it's broken, it seems like only one particular lookup key causes timeouts (and the EOFException every time) - other lookups work fine. I presume the index is corrupt somehow.
> {noformat}
> ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196) Exception in thread Thread[ReadStage:110,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:79)
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88)
> at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
> at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
> at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
> at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380)
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readFully(Unknown Source)
> at java.io.RandomAccessFile.readFully(Unknown Source)
> at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
> at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:110)
> at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)