You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Kjellman (JIRA)" <ji...@apache.org> on 2012/11/05 23:30:11 UTC
[jira] [Created] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Michael Kjellman created CASSANDRA-4916:
-------------------------------------------
Summary: Starting Cassandra throws EOF while reading saved cache
Key: CASSANDRA-4916
URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: Michael Kjellman
Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
at org.apache.cassandra.db.Table.initCf(Table.java:334)
at org.apache.cassandra.db.Table.<init>(Table.java:272)
at org.apache.cassandra.db.Table.open(Table.java:102)
at org.apache.cassandra.db.Table.open(Table.java:80)
at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Dave Brosius (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Brosius updated CASSANDRA-4916:
------------------------------------
Attachment: 4916.txt
oops fix formatting
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
> Attachments: 4916.txt
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4916:
--------------------------------------
Priority: Minor (was: Major)
Affects Version/s: (was: 1.2.0 beta 1)
Fix Version/s: 1.2.1
Assignee: Dave Brosius
Wouldn't be the first time I've seen InputStream.available lie. But we don't want to write the number of items in the cache at the start of the file (the approach we usually take) because that would require making a copy of the cache's keySet which might be more memory than we can afford.
Suggested workarounds: write some kind of EOF value when we're done instead of just closing the file, and check for that on read. Alternatively, just catch the EOF and log it at debug; a partially-read cache is harmless.
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Michael Kjellman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496291#comment-13496291 ]
Michael Kjellman commented on CASSANDRA-4916:
---------------------------------------------
[~jbellis] yes, will test now
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
> Attachments: 4916.txt
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495983#comment-13495983 ]
Jonathan Ellis commented on CASSANDRA-4916:
-------------------------------------------
Michael, can you test Dave's patch?
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
> Attachments: 4916.txt
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Dave Brosius (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Brosius updated CASSANDRA-4916:
------------------------------------
Attachment: (was: 4916.txt)
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
> Attachments: 4916.txt
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Michael Kjellman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496534#comment-13496534 ]
Michael Kjellman commented on CASSANDRA-4916:
---------------------------------------------
patch looks good. Ship it!
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
> Attachments: 4916.txt
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4916) Starting Cassandra throws EOF
while reading saved cache
Posted by "Dave Brosius (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Brosius updated CASSANDRA-4916:
------------------------------------
Attachment: 4916.txt
> Starting Cassandra throws EOF while reading saved cache
> -------------------------------------------------------
>
> Key: CASSANDRA-4916
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4916
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Michael Kjellman
> Assignee: Dave Brosius
> Priority: Minor
> Fix For: 1.2.1
>
> Attachments: 4916.txt
>
>
> Currently seeing nodes throw an EOF while reading a saved cache on the system schema when starting cassandra
> WARN 14:25:54,896 error reading saved cache /ssd/saved_caches/system-schema_columns-KeyCache-b.db
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:349)
> at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:378)
> at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:144)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:278)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:365)
> at org.apache.cassandra.db.Table.initCf(Table.java:334)
> at org.apache.cassandra.db.Table.<init>(Table.java:272)
> at org.apache.cassandra.db.Table.open(Table.java:102)
> at org.apache.cassandra.db.Table.open(Table.java:80)
> at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:320)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:203)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:395)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:438)
> to reproduce delete all data files, start a cluster, leave cluster up long enough to build a cache. nodetool drain, kill cassandra process. start cassandra process in foreground and note EOF thrown (see above for stack trace)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira