You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Michael Vaknine <mi...@citypath.com> on 2012/04/04 13:33:40 UTC
server down
I am recently having problems with 1 of my 4ty cluster servers.
I would appreciate any help
ERROR 11:23:14,331 Fatal exception in thread Thread[MutationStage:19,5,main]
225 java.lang.InternalError: a fault occurred in a recent unsafe memory
access operation in compiled Java code
226 at
org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.jav
a:196)
227 at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java
:348)
228 at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:1
20)
229 at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:8
3)
230 at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:7
3)
231 at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:3
7)
232 at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleS
liceReader.java:79)
233 at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleS
liceReader.java:40)
234 at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:140)
235 at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135
)
236 at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableS
liceIterator.java:107)
237 at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.jav
a:145)
238 at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.jav
a:124)
239 at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator
.java:98)
240 at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:140)
241 at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135
)
242 at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQ
ueryFilter.java:116)
243 at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:1
44)
244 at
org.apache.cassandra.db.CollationController.collectAllData(CollationControll
er.java:227)
245 at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont
roller.java:62)
246 at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto
re.java:1274)
247 at
org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:11
41)
248 at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1170)
249 at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1122)
250 at
org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504)
251 at org.apache.cassandra.db.Table.apply(Table.java:441)
252 at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:338
)
253 at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
254 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
255 at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
256 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
257 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
258 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
259 at java.lang.Thread.run(Thread.java:619)
260 INFO 11:23:17,640 Finished reading
/var/lib/cassandra/commitlog/CommitLog-1333465745624.log
261 ERROR 11:23:17,641 Exception encountered during startup
262 java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.InternalError: a fault occurred in a recent unsafe memory access
operation in compiled Java code
263 at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508)
Thanks
Michael
Re: server down
Posted by aaron morton <aa...@thelastpickle.com>.
Sun or Open JDK ?
Either way I would suggest upgrading to the latest JDK, upgrading cassandra to 1.0.8 and running nodetool upgradetables.
If the fault persists after that I would look at IO or memory issues.
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 5/04/2012, at 4:31 PM, Michael Vaknin wrote:
> i am usuinc cassandra 1.0.3 java 6.0.17 on ubuntu 10.04
> this is a stable version for 6 months now.
>
> On Wed, Apr 4, 2012 at 9:51 PM, aaron morton <aa...@thelastpickle.com> wrote:
> What version of cassandra are you using ?
>
> What java vendor / version ?
>
> What OS vendor / version ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 4/04/2012, at 11:33 PM, Michael Vaknine wrote:
>
>> I am recently having problems with 1 of my 4ty cluster servers.
>> I would appreciate any help
>>
>> ERROR 11:23:14,331 Fatal exception in thread Thread[MutationStage:19,5,main]
>> 225 java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
>> 226 at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
>> 227 at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:348)
>> 228 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
>> 229 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
>> 230 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
>> 231 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
>> 232 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
>> 233 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
>> 234 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>> 235 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>> 236 at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
>> 237 at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
>> 238 at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
>> 239 at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
>> 240 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>> 241 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>> 242 at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>> 243 at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
>> 244 at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:227)
>> 245 at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
>> 246 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1274)
>> 247 at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1141)
>> 248 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1170)
>> 249 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1122)
>> 250 at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504)
>> 251 at org.apache.cassandra.db.Table.apply(Table.java:441)
>> 252 at org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:338)
>> 253 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> 254 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> 255 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> 256 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> 257 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> 258 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> 259 at java.lang.Thread.run(Thread.java:619)
>> 260 INFO 11:23:17,640 Finished reading /var/lib/cassandra/commitlog/CommitLog-1333465745624.log
>> 261 ERROR 11:23:17,641 Exception encountered during startup
>> 262 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
>> 263 at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508)
>>
>> Thanks
>> Michael
>
>
Re: server down
Posted by Michael Vaknin <mi...@citypath.com>.
i am usuinc cassandra 1.0.3 java 6.0.17 on ubuntu 10.04
this is a stable version for 6 months now.
On Wed, Apr 4, 2012 at 9:51 PM, aaron morton <aa...@thelastpickle.com>wrote:
> What version of cassandra are you using ?
>
> What java vendor / version ?
>
> What OS vendor / version ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 4/04/2012, at 11:33 PM, Michael Vaknine wrote:
>
> I am recently having problems with 1 of my 4ty cluster servers.****
> I would appreciate any help****
> ** **
> ERROR 11:23:14,331 Fatal exception in thread
> Thread[MutationStage:19,5,main]****
> 225 java.lang.InternalError: a fault occurred in a recent unsafe memory
> access operation in compiled Java code****
> 226 at
> org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> ****
> 227 at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:348)
> ****
> 228 at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
> ****
> 229 at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
> ****
> 230 at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
> ****
> 231 at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
> ****
> 232 at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> ****
> 233 at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> ****
> 234 at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> ****
> 235 at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> ****
> 236 at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> ****
> 237 at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> ****
> 238 at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> ****
> 239 at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> ****
> 240 at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> ****
> 241 at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> ****
> 242 at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> ****
> 243 at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> ****
> 244 at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:227)
> ****
> 245 at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
> ****
> 246 at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1274)
> ****
> 247 at
> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1141)
> ****
> 248 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1170)
> ****
> 249 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1122)
> ****
> 250 at
> org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504)***
> *
> 251 at org.apache.cassandra.db.Table.apply(Table.java:441)****
> 252 at
> org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:338)
> ****
> 253 at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)***
> *
> 254 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)***
> *
> 255 at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)****
> 256 at java.util.concurrent.FutureTask.run(FutureTask.java:138)****
> 257 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ****
> 258 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ****
> 259 at java.lang.Thread.run(Thread.java:619)****
> 260 INFO 11:23:17,640 Finished reading
> /var/lib/cassandra/commitlog/CommitLog-1333465745624.log****
> 261 ERROR 11:23:17,641 Exception encountered during startup****
> 262 java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.InternalError: a fault occurred in a recent unsafe memory access
> operation in compiled Java code****
> 263 at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508)
> ****
> ** **
> Thanks****
> Michael****
>
>
>
Re: server down
Posted by aaron morton <aa...@thelastpickle.com>.
What version of cassandra are you using ?
What java vendor / version ?
What OS vendor / version ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 4/04/2012, at 11:33 PM, Michael Vaknine wrote:
> I am recently having problems with 1 of my 4ty cluster servers.
> I would appreciate any help
>
> ERROR 11:23:14,331 Fatal exception in thread Thread[MutationStage:19,5,main]
> 225 java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
> 226 at org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196)
> 227 at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:348)
> 228 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
> 229 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
> 230 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
> 231 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
> 232 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> 233 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> 234 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> 235 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> 236 at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
> 237 at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> 238 at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124)
> 239 at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> 240 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> 241 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> 242 at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> 243 at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144)
> 244 at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:227)
> 245 at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
> 246 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1274)
> 247 at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1141)
> 248 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1170)
> 249 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1122)
> 250 at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504)
> 251 at org.apache.cassandra.db.Table.apply(Table.java:441)
> 252 at org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:338)
> 253 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 254 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> 255 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 256 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 257 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 258 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 259 at java.lang.Thread.run(Thread.java:619)
> 260 INFO 11:23:17,640 Finished reading /var/lib/cassandra/commitlog/CommitLog-1333465745624.log
> 261 ERROR 11:23:17,641 Exception encountered during startup
> 262 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
> 263 at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508)
>
> Thanks
> Michael