You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by dragos cernahoschi <dr...@gmail.com> on 2010/10/29 17:38:24 UTC
more info CASSANDRA-1472
The stress.py command:
1. python contrib/py_stress/stress.py -C 32 -x keys_bitmap
2. more info from stack trace:
10/10/29 18:15:28 INFO db.Memtable: Writing
Memtable-Standard1@23048841(17806140
bytes, 349140 operations)
10/10/29 18:15:29 INFO service.GCInspector: GC for PS MarkSweep: 789 ms,
7178512 reclaimed leaving 320182640 used; max is 1383202816
10/10/29 18:15:31 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:1,5,main]
java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:177)
at
org.apache.avro.generic.GenericDatumReader.readEnum(GenericDatumReader.java:172)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:115)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:118)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
at
org.apache.cassandra.io.SerDeUtils.deserializeWithSchema(SerDeUtils.java:112)
at
org.apache.cassandra.io.sstable.bitidx.BitmapIndexReader.open(BitmapIndexReader.java:87)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:196)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:178)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:160)
at org.apache.cassandra.db.Memtable.access$1(Memtable.java:152)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:172)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
10/10/29 18:15:38 INFO service.GCInspector: GC for PS MarkSweep: 896 ms,
72939816 reclaimed leaving 341851792 used; max is 1359740928
10/10/29 18:15:38 INFO service.GCInspector: GC for PS Scavenge: 312 ms,
188201560 reclaimed leaving 414791608 used; max is 1359740928
10/10/29 18:15:40 INFO db.ColumnFamilyStore: switching in a fresh Memtable
for Standard1 at
CommitLogContext(file='/home/dragos/cassandra/commitlog/CommitLog-1288365280350.log',
position=111458032)
10/10/29 18:15:40 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Standard1@25762909(17806140 bytes, 349140 operations)
10/10/29 18:15:40 INFO db.Memtable: Writing
Memtable-Standard1@25762909(17806140
bytes, 349140 operations)
10/10/29 18:15:43 INFO service.GCInspector: GC for PS MarkSweep: 1309 ms,
293768 reclaimed leaving 456751520 used; max is 1349451776
10/10/29 18:15:43 INFO service.GCInspector: Pool Name
Active Pending
10/10/29 18:15:43 INFO service.GCInspector:
ResponseStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
ReadStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
ReadRepair 0 0
10/10/29 18:15:43 INFO service.GCInspector: MutationStage
32 49
10/10/29 18:15:43 INFO service.GCInspector:
GossipStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
AntientropyStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
MigrationStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
StreamStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
MemtablePostFlusher 1 3
10/10/29 18:15:43 INFO service.GCInspector:
FlushWriter 1 1
10/10/29 18:15:43 INFO service.GCInspector:
MiscStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
FlushSorter 0 0
10/10/29 18:15:43 INFO service.GCInspector: CompactionManager
n/a 0
10/10/29 18:15:43 INFO service.GCInspector: MessagingService
n/a 0,0
10/10/29 18:15:43 INFO service.GCInspector: ColumnFamily
Memtable ops,data Row cache size/cap Key cache size/cap
10/10/29 18:15:43 INFO service.GCInspector:
Keyspace1.Super1 0,0 0/0
0/200000
10/10/29 18:15:43 INFO service.GCInspector: Keyspace1.Standard1
67080,3421080 0/0 0/200000
10/10/29 18:15:43 INFO service.GCInspector:
system.LocationInfo 1,17
0/0 1/2
10/10/29 18:15:43 INFO service.GCInspector:
system.HintsColumnFamily 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector:
system.Migrations 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector:
system.Schema 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector:
system.IndexInfo 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector: GC for PS Scavenge: 398 ms,
234385872 reclaimed leaving 457045288 used; max is 1349451776
Re: more info CASSANDRA-1472
Posted by dragos cernahoschi <dr...@gmail.com>.
Patches applied. Exception disappeared.
On Mon, Nov 1, 2010 at 7:37 PM, Stu Hood <st...@rackspace.com> wrote:
> trunk @ r1029546
>
> -----Original Message-----
> From: "dragos cernahoschi" <dr...@gmail.com>
> Sent: Monday, November 1, 2010 12:20pm
> To: dev@cassandra.apache.org
> Subject: Re: more info CASSANDRA-1472
>
> I can't apply the patches on the latest trunk. Which trunk version
> (checksum) did you rebased?
>
> Thank you!
>
> On Mon, Nov 1, 2010 at 6:50 AM, Stu Hood <st...@rackspace.com> wrote:
>
> > I can't reproduce this, but I've posted a rebased version on #1472 that
> you
> > can try out. Thanks for trying it out!
> >
> >
> > -----Original Message-----
> > From: "dragos cernahoschi" <dr...@gmail.com>
> > Sent: Friday, October 29, 2010 10:38am
> > To: dev@cassandra.apache.org
> > Subject: more info CASSANDRA-1472
> >
> > The stress.py command:
> >
> > 1. python contrib/py_stress/stress.py -C 32 -x keys_bitmap
> >
> > 2. more info from stack trace:
> >
> > 10/10/29 18:15:28 INFO db.Memtable: Writing
> > Memtable-Standard1@23048841(17806140
> > bytes, 349140 operations)
> > 10/10/29 18:15:29 INFO service.GCInspector: GC for PS MarkSweep: 789 ms,
> > 7178512 reclaimed leaving 320182640 used; max is 1383202816
> > 10/10/29 18:15:31 ERROR service.AbstractCassandraDaemon: Fatal exception
> in
> > thread Thread[FlushWriter:1,5,main]
> > java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 1
> > at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> > at
> > org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:177)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.readEnum(GenericDatumReader.java:172)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:115)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:118)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
> > at
> >
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
> > at
> >
> >
> org.apache.cassandra.io.SerDeUtils.deserializeWithSchema(SerDeUtils.java:112)
> > at
> >
> >
> org.apache.cassandra.io.sstable.bitidx.BitmapIndexReader.open(BitmapIndexReader.java:87)
> > at
> >
> >
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:196)
> > at
> >
> >
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:178)
> > at
> > org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:160)
> > at org.apache.cassandra.db.Memtable.access$1(Memtable.java:152)
> > at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:172)
> > at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > ... 6 more
> > 10/10/29 18:15:38 INFO service.GCInspector: GC for PS MarkSweep: 896 ms,
> > 72939816 reclaimed leaving 341851792 used; max is 1359740928
> > 10/10/29 18:15:38 INFO service.GCInspector: GC for PS Scavenge: 312 ms,
> > 188201560 reclaimed leaving 414791608 used; max is 1359740928
> > 10/10/29 18:15:40 INFO db.ColumnFamilyStore: switching in a fresh
> Memtable
> > for Standard1 at
> >
> >
> CommitLogContext(file='/home/dragos/cassandra/commitlog/CommitLog-1288365280350.log',
> > position=111458032)
> > 10/10/29 18:15:40 INFO db.ColumnFamilyStore: Enqueuing flush of
> > Memtable-Standard1@25762909(17806140 bytes, 349140 operations)
> > 10/10/29 18:15:40 INFO db.Memtable: Writing
> > Memtable-Standard1@25762909(17806140
> > bytes, 349140 operations)
> > 10/10/29 18:15:43 INFO service.GCInspector: GC for PS MarkSweep: 1309 ms,
> > 293768 reclaimed leaving 456751520 used; max is 1349451776
> > 10/10/29 18:15:43 INFO service.GCInspector: Pool Name
> > Active Pending
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > ResponseStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > ReadStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > ReadRepair 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector: MutationStage
> > 32 49
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > GossipStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > AntientropyStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > MigrationStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > StreamStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > MemtablePostFlusher 1 3
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > FlushWriter 1 1
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > MiscStage 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > FlushSorter 0 0
> > 10/10/29 18:15:43 INFO service.GCInspector: CompactionManager
> > n/a 0
> > 10/10/29 18:15:43 INFO service.GCInspector: MessagingService
> > n/a 0,0
> > 10/10/29 18:15:43 INFO service.GCInspector: ColumnFamily
> > Memtable ops,data Row cache size/cap Key cache size/cap
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > Keyspace1.Super1 0,0 0/0
> > 0/200000
> > 10/10/29 18:15:43 INFO service.GCInspector: Keyspace1.Standard1
> > 67080,3421080 0/0 0/200000
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > system.LocationInfo 1,17
> > 0/0 1/2
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > system.HintsColumnFamily 0,0
> > 0/0 0/1
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > system.Migrations 0,0
> > 0/0 0/1
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > system.Schema 0,0
> > 0/0 0/1
> > 10/10/29 18:15:43 INFO service.GCInspector:
> > system.IndexInfo 0,0
> > 0/0 0/1
> > 10/10/29 18:15:43 INFO service.GCInspector: GC for PS Scavenge: 398 ms,
> > 234385872 reclaimed leaving 457045288 used; max is 1349451776
> >
> >
> >
>
>
>
Re: more info CASSANDRA-1472
Posted by Stu Hood <st...@rackspace.com>.
trunk @ r1029546
-----Original Message-----
From: "dragos cernahoschi" <dr...@gmail.com>
Sent: Monday, November 1, 2010 12:20pm
To: dev@cassandra.apache.org
Subject: Re: more info CASSANDRA-1472
I can't apply the patches on the latest trunk. Which trunk version
(checksum) did you rebased?
Thank you!
On Mon, Nov 1, 2010 at 6:50 AM, Stu Hood <st...@rackspace.com> wrote:
> I can't reproduce this, but I've posted a rebased version on #1472 that you
> can try out. Thanks for trying it out!
>
>
> -----Original Message-----
> From: "dragos cernahoschi" <dr...@gmail.com>
> Sent: Friday, October 29, 2010 10:38am
> To: dev@cassandra.apache.org
> Subject: more info CASSANDRA-1472
>
> The stress.py command:
>
> 1. python contrib/py_stress/stress.py -C 32 -x keys_bitmap
>
> 2. more info from stack trace:
>
> 10/10/29 18:15:28 INFO db.Memtable: Writing
> Memtable-Standard1@23048841(17806140
> bytes, 349140 operations)
> 10/10/29 18:15:29 INFO service.GCInspector: GC for PS MarkSweep: 789 ms,
> 7178512 reclaimed leaving 320182640 used; max is 1383202816
> 10/10/29 18:15:31 ERROR service.AbstractCassandraDaemon: Fatal exception in
> thread Thread[FlushWriter:1,5,main]
> java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:177)
> at
>
> org.apache.avro.generic.GenericDatumReader.readEnum(GenericDatumReader.java:172)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:115)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:118)
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
> at
>
> org.apache.cassandra.io.SerDeUtils.deserializeWithSchema(SerDeUtils.java:112)
> at
>
> org.apache.cassandra.io.sstable.bitidx.BitmapIndexReader.open(BitmapIndexReader.java:87)
> at
>
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:196)
> at
>
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:178)
> at
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:160)
> at org.apache.cassandra.db.Memtable.access$1(Memtable.java:152)
> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:172)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
> 10/10/29 18:15:38 INFO service.GCInspector: GC for PS MarkSweep: 896 ms,
> 72939816 reclaimed leaving 341851792 used; max is 1359740928
> 10/10/29 18:15:38 INFO service.GCInspector: GC for PS Scavenge: 312 ms,
> 188201560 reclaimed leaving 414791608 used; max is 1359740928
> 10/10/29 18:15:40 INFO db.ColumnFamilyStore: switching in a fresh Memtable
> for Standard1 at
>
> CommitLogContext(file='/home/dragos/cassandra/commitlog/CommitLog-1288365280350.log',
> position=111458032)
> 10/10/29 18:15:40 INFO db.ColumnFamilyStore: Enqueuing flush of
> Memtable-Standard1@25762909(17806140 bytes, 349140 operations)
> 10/10/29 18:15:40 INFO db.Memtable: Writing
> Memtable-Standard1@25762909(17806140
> bytes, 349140 operations)
> 10/10/29 18:15:43 INFO service.GCInspector: GC for PS MarkSweep: 1309 ms,
> 293768 reclaimed leaving 456751520 used; max is 1349451776
> 10/10/29 18:15:43 INFO service.GCInspector: Pool Name
> Active Pending
> 10/10/29 18:15:43 INFO service.GCInspector:
> ResponseStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> ReadStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> ReadRepair 0 0
> 10/10/29 18:15:43 INFO service.GCInspector: MutationStage
> 32 49
> 10/10/29 18:15:43 INFO service.GCInspector:
> GossipStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> AntientropyStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> MigrationStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> StreamStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> MemtablePostFlusher 1 3
> 10/10/29 18:15:43 INFO service.GCInspector:
> FlushWriter 1 1
> 10/10/29 18:15:43 INFO service.GCInspector:
> MiscStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> FlushSorter 0 0
> 10/10/29 18:15:43 INFO service.GCInspector: CompactionManager
> n/a 0
> 10/10/29 18:15:43 INFO service.GCInspector: MessagingService
> n/a 0,0
> 10/10/29 18:15:43 INFO service.GCInspector: ColumnFamily
> Memtable ops,data Row cache size/cap Key cache size/cap
> 10/10/29 18:15:43 INFO service.GCInspector:
> Keyspace1.Super1 0,0 0/0
> 0/200000
> 10/10/29 18:15:43 INFO service.GCInspector: Keyspace1.Standard1
> 67080,3421080 0/0 0/200000
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.LocationInfo 1,17
> 0/0 1/2
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.HintsColumnFamily 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.Migrations 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.Schema 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.IndexInfo 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector: GC for PS Scavenge: 398 ms,
> 234385872 reclaimed leaving 457045288 used; max is 1349451776
>
>
>
Re: more info CASSANDRA-1472
Posted by dragos cernahoschi <dr...@gmail.com>.
I can't apply the patches on the latest trunk. Which trunk version
(checksum) did you rebased?
Thank you!
On Mon, Nov 1, 2010 at 6:50 AM, Stu Hood <st...@rackspace.com> wrote:
> I can't reproduce this, but I've posted a rebased version on #1472 that you
> can try out. Thanks for trying it out!
>
>
> -----Original Message-----
> From: "dragos cernahoschi" <dr...@gmail.com>
> Sent: Friday, October 29, 2010 10:38am
> To: dev@cassandra.apache.org
> Subject: more info CASSANDRA-1472
>
> The stress.py command:
>
> 1. python contrib/py_stress/stress.py -C 32 -x keys_bitmap
>
> 2. more info from stack trace:
>
> 10/10/29 18:15:28 INFO db.Memtable: Writing
> Memtable-Standard1@23048841(17806140
> bytes, 349140 operations)
> 10/10/29 18:15:29 INFO service.GCInspector: GC for PS MarkSweep: 789 ms,
> 7178512 reclaimed leaving 320182640 used; max is 1383202816
> 10/10/29 18:15:31 ERROR service.AbstractCassandraDaemon: Fatal exception in
> thread Thread[FlushWriter:1,5,main]
> java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:177)
> at
>
> org.apache.avro.generic.GenericDatumReader.readEnum(GenericDatumReader.java:172)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:115)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:118)
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
> at
>
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
> at
>
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
> at
>
> org.apache.cassandra.io.SerDeUtils.deserializeWithSchema(SerDeUtils.java:112)
> at
>
> org.apache.cassandra.io.sstable.bitidx.BitmapIndexReader.open(BitmapIndexReader.java:87)
> at
>
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:196)
> at
>
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:178)
> at
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:160)
> at org.apache.cassandra.db.Memtable.access$1(Memtable.java:152)
> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:172)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
> 10/10/29 18:15:38 INFO service.GCInspector: GC for PS MarkSweep: 896 ms,
> 72939816 reclaimed leaving 341851792 used; max is 1359740928
> 10/10/29 18:15:38 INFO service.GCInspector: GC for PS Scavenge: 312 ms,
> 188201560 reclaimed leaving 414791608 used; max is 1359740928
> 10/10/29 18:15:40 INFO db.ColumnFamilyStore: switching in a fresh Memtable
> for Standard1 at
>
> CommitLogContext(file='/home/dragos/cassandra/commitlog/CommitLog-1288365280350.log',
> position=111458032)
> 10/10/29 18:15:40 INFO db.ColumnFamilyStore: Enqueuing flush of
> Memtable-Standard1@25762909(17806140 bytes, 349140 operations)
> 10/10/29 18:15:40 INFO db.Memtable: Writing
> Memtable-Standard1@25762909(17806140
> bytes, 349140 operations)
> 10/10/29 18:15:43 INFO service.GCInspector: GC for PS MarkSweep: 1309 ms,
> 293768 reclaimed leaving 456751520 used; max is 1349451776
> 10/10/29 18:15:43 INFO service.GCInspector: Pool Name
> Active Pending
> 10/10/29 18:15:43 INFO service.GCInspector:
> ResponseStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> ReadStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> ReadRepair 0 0
> 10/10/29 18:15:43 INFO service.GCInspector: MutationStage
> 32 49
> 10/10/29 18:15:43 INFO service.GCInspector:
> GossipStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> AntientropyStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> MigrationStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> StreamStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> MemtablePostFlusher 1 3
> 10/10/29 18:15:43 INFO service.GCInspector:
> FlushWriter 1 1
> 10/10/29 18:15:43 INFO service.GCInspector:
> MiscStage 0 0
> 10/10/29 18:15:43 INFO service.GCInspector:
> FlushSorter 0 0
> 10/10/29 18:15:43 INFO service.GCInspector: CompactionManager
> n/a 0
> 10/10/29 18:15:43 INFO service.GCInspector: MessagingService
> n/a 0,0
> 10/10/29 18:15:43 INFO service.GCInspector: ColumnFamily
> Memtable ops,data Row cache size/cap Key cache size/cap
> 10/10/29 18:15:43 INFO service.GCInspector:
> Keyspace1.Super1 0,0 0/0
> 0/200000
> 10/10/29 18:15:43 INFO service.GCInspector: Keyspace1.Standard1
> 67080,3421080 0/0 0/200000
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.LocationInfo 1,17
> 0/0 1/2
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.HintsColumnFamily 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.Migrations 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.Schema 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector:
> system.IndexInfo 0,0
> 0/0 0/1
> 10/10/29 18:15:43 INFO service.GCInspector: GC for PS Scavenge: 398 ms,
> 234385872 reclaimed leaving 457045288 used; max is 1349451776
>
>
>
RE: more info CASSANDRA-1472
Posted by Stu Hood <st...@rackspace.com>.
I can't reproduce this, but I've posted a rebased version on #1472 that you can try out. Thanks for trying it out!
-----Original Message-----
From: "dragos cernahoschi" <dr...@gmail.com>
Sent: Friday, October 29, 2010 10:38am
To: dev@cassandra.apache.org
Subject: more info CASSANDRA-1472
The stress.py command:
1. python contrib/py_stress/stress.py -C 32 -x keys_bitmap
2. more info from stack trace:
10/10/29 18:15:28 INFO db.Memtable: Writing
Memtable-Standard1@23048841(17806140
bytes, 349140 operations)
10/10/29 18:15:29 INFO service.GCInspector: GC for PS MarkSweep: 789 ms,
7178512 reclaimed leaving 320182640 used; max is 1383202816
10/10/29 18:15:31 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:1,5,main]
java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:177)
at
org.apache.avro.generic.GenericDatumReader.readEnum(GenericDatumReader.java:172)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:115)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:118)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:142)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:114)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
at
org.apache.cassandra.io.SerDeUtils.deserializeWithSchema(SerDeUtils.java:112)
at
org.apache.cassandra.io.sstable.bitidx.BitmapIndexReader.open(BitmapIndexReader.java:87)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:196)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:178)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:160)
at org.apache.cassandra.db.Memtable.access$1(Memtable.java:152)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:172)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
10/10/29 18:15:38 INFO service.GCInspector: GC for PS MarkSweep: 896 ms,
72939816 reclaimed leaving 341851792 used; max is 1359740928
10/10/29 18:15:38 INFO service.GCInspector: GC for PS Scavenge: 312 ms,
188201560 reclaimed leaving 414791608 used; max is 1359740928
10/10/29 18:15:40 INFO db.ColumnFamilyStore: switching in a fresh Memtable
for Standard1 at
CommitLogContext(file='/home/dragos/cassandra/commitlog/CommitLog-1288365280350.log',
position=111458032)
10/10/29 18:15:40 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Standard1@25762909(17806140 bytes, 349140 operations)
10/10/29 18:15:40 INFO db.Memtable: Writing
Memtable-Standard1@25762909(17806140
bytes, 349140 operations)
10/10/29 18:15:43 INFO service.GCInspector: GC for PS MarkSweep: 1309 ms,
293768 reclaimed leaving 456751520 used; max is 1349451776
10/10/29 18:15:43 INFO service.GCInspector: Pool Name
Active Pending
10/10/29 18:15:43 INFO service.GCInspector:
ResponseStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
ReadStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
ReadRepair 0 0
10/10/29 18:15:43 INFO service.GCInspector: MutationStage
32 49
10/10/29 18:15:43 INFO service.GCInspector:
GossipStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
AntientropyStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
MigrationStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
StreamStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
MemtablePostFlusher 1 3
10/10/29 18:15:43 INFO service.GCInspector:
FlushWriter 1 1
10/10/29 18:15:43 INFO service.GCInspector:
MiscStage 0 0
10/10/29 18:15:43 INFO service.GCInspector:
FlushSorter 0 0
10/10/29 18:15:43 INFO service.GCInspector: CompactionManager
n/a 0
10/10/29 18:15:43 INFO service.GCInspector: MessagingService
n/a 0,0
10/10/29 18:15:43 INFO service.GCInspector: ColumnFamily
Memtable ops,data Row cache size/cap Key cache size/cap
10/10/29 18:15:43 INFO service.GCInspector:
Keyspace1.Super1 0,0 0/0
0/200000
10/10/29 18:15:43 INFO service.GCInspector: Keyspace1.Standard1
67080,3421080 0/0 0/200000
10/10/29 18:15:43 INFO service.GCInspector:
system.LocationInfo 1,17
0/0 1/2
10/10/29 18:15:43 INFO service.GCInspector:
system.HintsColumnFamily 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector:
system.Migrations 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector:
system.Schema 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector:
system.IndexInfo 0,0
0/0 0/1
10/10/29 18:15:43 INFO service.GCInspector: GC for PS Scavenge: 398 ms,
234385872 reclaimed leaving 457045288 used; max is 1349451776