[jira] [Comment Edited] (CASSANDRA-11014) Repair fails with "not enough bytes"

["Marcus Eriksson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106797#comment-15106797 ] 

Marcus Eriksson edited comment on CASSANDRA-11014 at 1/19/16 2:36 PM:
----------------------------------------------------------------------

Is it possible for you to identify the sstable that is broken? I think there should be a 'Scrubbing ....' log message before the error in scrub-output.txt. Could you attach it if so? We would also need the schema


was (Author: krummas):
Is it possible for you to identify the sstable that is broken? I think there should be a 'Scrubbing ....' log message before the error in scrub-output.txt. Could you attach it if so? 

> Repair fails with "not enough bytes"
> ------------------------------------
>
>                 Key: CASSANDRA-11014
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11014
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 3 node cluster, debian jessie, cassandra 2.2.4
>            Reporter: Christian Schjolberg
>            Priority: Blocker
>         Attachments: scrub-output.txt
>
>
> After upgrading to 2.2.4, nodetool repair fails every time with the error message "Not enough bytes". It appears no data is being repaired at all. Here's some output:
> -@cas01:~$ nodetool repair
> [2016-01-14 12:00:16,590] Starting repair command #1, repairing keyspace adsquare with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 768)
> [2016-01-14 12:00:21,935] Repair session 61174f80-bab6-11e5-9fa9-11175757c857 for range (-3942884673882176939,-3929110923969659376] failed with error [repair #61174f80-bab6-11e5-9fa9-11175757c857 on adsquare/device_lookup, (-3942884673882176939,-3929110923969659376]] Validation failed in /10.10.100.61 (progress: 0%)
> The system.log on the host in question shows 
> ERROR [ValidationExecutor:2] 2016-01-14 09:58:19,935 CassandraDaemon.java:185 - Exception in thread Thread[ValidationExecutor:2,1,main]
> java.lang.IllegalArgumentException: Not enough bytes
>         at org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
>         at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:169) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
>         at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645) ~[guava-16.0.jar:na]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.update(LazilyCompactedRow.java:172) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.repair.Validator.rowHash(Validator.java:194) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.repair.Validator.add(Validator.java:143) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1118) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:73) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:671) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_66]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
> Thinking that there might be some corruption I ran nodetool scrub, which fills the log with statements like the one in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)