You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chakravarthi Manepalli (JIRA)" <ji...@apache.org> on 2019/08/06 11:23:00 UTC
[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900912#comment-16900912 ]
Chakravarthi Manepalli commented on CASSANDRA-13556:
----------------------------------------------------
I have observed the same issue in one of our setups. There were no removal or addition of columns in a recent period. I have observed the stack error in debug log.
WARN [ReadStage-1] 2019-08-06 16:34:29,093 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {}
java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /media/db_store/data/server_data/known_ap_ssid_list-d2cde2f0adf211e99625cd7f2acaf155/mc-5-big-Data.db
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2598) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /media/db_store/data/server_data/known_ap_ssid_list-d2cde2f0adf211e99625cd7f2acaf155/mc-5-big-Data.db
at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:391) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:258) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:116) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1886) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2594) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 5 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: null
So I tried to restart the server and this is in my startup log
ERROR [CompactionExecutor:2] 2019-08-06 16:38:08,930 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:2,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /media/db_store/data/server_data/known_ap_ssid_list-d2cde2f0adf211e99625cd7f2acaf155/mc-5-big-Data.db
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:127) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:110) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Caused by: java.lang.ArrayIndexOutOfBoundsException: 8
at org.apache.cassandra.db.ClusteringBoundOrBoundary$Serializer.deserialize(ClusteringBoundOrBoundary.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:470) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:436) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:95) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:73) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 27 common frames omitted
ERROR [CompactionExecutor:2] 2019-08-06 16:38:08,935 DefaultFSErrorHandler.java:92 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /media/db_store/data/server_data/known_ap_ssid_list-d2cde2f0adf211e99625cd7f2acaf155/mc-5-big-Data.db
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:127) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:499) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:359) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:110) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_152]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Caused by: java.lang.ArrayIndexOutOfBoundsException: 8
at org.apache.cassandra.db.ClusteringBoundOrBoundary$Serializer.deserialize(ClusteringBoundOrBoundary.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:470) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:436) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:95) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:73) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 27 common frames omitted
I deleted the corrupt file and restarted my node and it is working fine.
> Corrupted SSTables
> ------------------
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
> Reporter: Ihor Prokopov
> Priority: Normal
> Fix For: 3.11.x
>
>
> After 3 month of working, we noticed that number of compaction tasks were growing (~600 pending tasks). SStables verification shows that some of them were corrupted. Repairing didn't help (it was crashing with error).
> Also some of requests (f.e. select * from fetcher where domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next error:
> {color:red}
> Traceback (most recent call last):
> File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in perform_simple_statement
> result = future.result()
> File "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py", line 3650, in result
> raise self._final_exception
> error: unpack requires a string argument of length 4
> {color}
> Table chema:
> {quote}
> CREATE TABLE fetcher (
> domain bigint,
> uri bigint,
> date date,
> content_length int,
> elapsed float,
> encoding text,
> fetched_time bigint,
> flinks frozen<set<tuple<bigint, bigint»>,
> flinks_count int,
> html_fingerprint bigint,
> indexed boolean,
> adult boolean,
> kws_count int,
> lang_id int,
> last_updated bigint,
> redirect_url tuple<bigint, bigint>,
> revisit_date date,
> revisit_interval int,
> status_code int,
> tokens_fingerprint bigint,
> uris frozen<set<bigint»,
> url text,
> PRIMARY KEY (domain, uri, date)
> ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'fetcher history'
> AND compaction = \{'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
> 'sstable_size_in_mb': '256',
> 'tombstone_threshold': '.2'}
> AND compression = \{'chunk_length_in_kb': '64',
> 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.5
> AND speculative_retry = '99PERCENTILE';
> {quote}
> Corrupted [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing].
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org