You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Michael Vaknine <mi...@citypath.com> on 2011/11/13 11:35:11 UTC
Upgrade Cassandra Cluster to 1.0.2
Hi,
I would appreciate any help.
I have a cluster of 4 servers with replication factor 3 version 1.0.0
The cluster was upgraded from 0.7.8.
I am trying to upgrade to 1.0.2 and when I try to start the first upgraded
server I get the following error
ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[WRITE-/10.5.6.102,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou
ndTcpConnectionPool.java:93)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn
ectionPool.java:77)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection
.java:209)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:93)
ERROR [WRITE-/10.4.251.82] 2011-11-13 10:20:37,448
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[WRITE-/10.4.251.82,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou
ndTcpConnectionPool.java:93)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn
ectionPool.java:77)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection
.java:209)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:93)
ERROR [WRITE-/10.64.146.54] 2011-11-13 10:20:37,449
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[WRITE-/10.64.146.54,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou
ndTcpConnectionPool.java:93)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn
ectionPool.java:77)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection
.java:209)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:93)
when I run nodetool version I get
ReleaseVersion: 1.0.2
When I run nodetool ring I get
Address DC Rack Status State Load Owns
Token
127605887595351923798765477786913079296
10.5.6.102 datacenter1 rack1 Down Normal ?
25.00% 0
10.4.251.82 datacenter1 rack1 Down Normal ?
25.00% 42535295865117307932921825928971026432
10.64.146.54 datacenter1 rack1 Down Normal ?
25.00% 85070591730234615865843651857942052864
10.104.121.246 datacenter1 rack1 Up Normal 9.55 GB
25.00% 127605887595351923798765477786913079296
The only server up is the one I have upgraded.
When I run nodetool ring from a different server on the cluster I get this
10.5.6.102 datacenter1 rack1 Up Normal 22.56 GB
25.00% 0
10.4.251.82 datacenter1 rack1 Up Normal 18.85 GB
25.00% 42535295865117307932921825928971026432
10.64.146.54 datacenter1 rack1 Up Normal 16.24 GB
25.00% 85070591730234615865843651857942052864
10.104.121.246 datacenter1 rack1 Down Normal 10.21 GB
25.00% 127605887595351923798765477786913079296
All up except for the upgraded server.
Thanks
Michael vaknine
Upgrade cassandra to 1.0.0
Posted by Michael Vaknine <mi...@citypath.com>.
I am upgrading Cassandra 0.7.8 to 1.0.0 got error
ERROR [SSTableBatchOpen:2] 2011-11-22 09:48:00,000
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[SSTableBatchOpen:2,5,main],
java.lang.AssertionError,
org.apache.cassandra.io.sstable.SSTable.<init>(SSTable.java:99),
org.apache.cassandra.io.sstable.SSTableReader.<init>(SSTableReader.java:261)
,
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:139),
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:196),
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441),
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303),
java.util.concurrent.FutureTask.run(FutureTask.java:138),
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886),
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08),
java.lang.Thread.run(Thread.java:619)
Thanks
Michael
Re: Upgrade Cassandra Cluster to 1.0.3
Posted by Jonathan Ellis <jb...@gmail.com>.
That should do the trick.
2011/11/23 Michael Vaknine <mi...@citypath.com>:
> Hi Jonathan,
>
> You are right I had 1 node 1.0.2 for some reason so I did the upgrade again.
> I have now a 4 cluster upgraded to 1.0.3 but now I get the following error
> on 2 nodes on the cluster:
>
> ERROR [HintedHandoff:3] 2011-11-23 06:39:31,250 AbstractCassandraDaemon.java
> (line 133) Fatal exception in thread Thread[HintedHandoff:3,1,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHa
> ndOffManager.java:301)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager
> .java:81)
> at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffMana
> ger.java:353)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
> at java.lang.Thread.run(Thread.java:619)
>
> if I did cassandra repair on all the servers on the cluster can I simply
> truncate the column family HintsColumnFamily?
> Will I lose any data?
>
> Thanks
> Michael
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Monday, November 21, 2011 8:40 PM
> To: user@cassandra.apache.org
> Cc: cassandra-user@incubator.apache.org
> Subject: Re: Upgrade Cassandra Cluster to 1.0.3
>
> Sounds to me like
> https://issues.apache.org/jira/browse/CASSANDRA-3491, which was
> present in 1.0.2 and fixed in 1.0.3. It sounds like you're running
> the wrong version by mistake.
>
> On Mon, Nov 21, 2011 at 7:53 AM, Michael Vaknine <mi...@citypath.com>
> wrote:
>> Hi,
>> Any help will be appreciated.
>>
>> I am upgrading Cassandra 1.0.0 to 1.0.3 got error
>> ERROR [CompactionExecutor:3] 2011-11-21 11:10:59,075
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor: 3,1,main]
>> 3645 java.lang.StackOverflowError
>> 3646 at com.google.common.base.Objects.equal(Objects.java:51)
>> 3647 at org.apache.cassandra.utils.Pair.equals(Pair.java:48)
>> 3648 at
>>
> java.util.concurrent.ConcurrentHashMap$Segment.get(ConcurrentHashMap.java:33
>> 8)
>> 3649 at
>> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:769)
>> 3650 at
>>
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.get(Concurren
>> tLinkedHashMap.java:740)
>> 3651 at
>>
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.get(ConcurrentLinkedHas
>> hCache.java:81)
>> 3652 at
>>
> org.apache.cassandra.cache.InstrumentingCache.get(InstrumentingCache.java:68
>> )
>> 3653 at
>>
> org.apache.cassandra.io.sstable.SSTableReader.getCachedPosition(SSTableReade
>> r.java:598)
>> 3654 at
>>
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java
>> :621)
>> 3655 at
>>
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader
>> .java:786)
>> 3656 at
>>
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNa
>> mesIterator.java:61)
>>
>> I tried to increase Xss to 640M and then got this error
>>
>> ERROR [Thread-28] 2011-11-21 12:52:40,808 AbstractCassandraDaemon.java
> (line
>> 133) Fatal exception in thread Thread[Thread-28,5,main]
>> 4940 java.lang.RuntimeException: java.util.concurrent.ExecutionException:
>> java.lang.Error: Maximum lock count exceeded
>> 4941 at
>>
> org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
>> es(SecondaryIndexManager.java:131)
>> 4942 at
>>
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSessi
>> on.java:151)
>> 4943 at
>>
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReade
>> r.java:102)
>> 4944 at
>>
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.
>> java:184)
>> 4945 at
>>
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
>> a:81)
>> 4946 Caused by: java.util.concurrent.ExecutionException: java.lang.Error:
>> Maximum lock count exceeded
>> 4947 at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> 4948 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> 4949 at
>>
> org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
>> es(SecondaryIndexManager.java:122)
>> 4950 ... 4 more
>> 4951 Caused by: java.lang.Error: Maximum lock count exceeded
>>
>> None of this errors happen on 1.0.0
>>
>>
>> Thanks
>> Michael
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
RE: Upgrade Cassandra Cluster to 1.0.3
Posted by Michael Vaknine <mi...@citypath.com>.
Hi Jonathan,
You are right I had 1 node 1.0.2 for some reason so I did the upgrade again.
I have now a 4 cluster upgraded to 1.0.3 but now I get the following error
on 2 nodes on the cluster:
ERROR [HintedHandoff:3] 2011-11-23 06:39:31,250 AbstractCassandraDaemon.java
(line 133) Fatal exception in thread Thread[HintedHandoff:3,1,main]
java.lang.AssertionError
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHa
ndOffManager.java:301)
at
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager
.java:81)
at
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffMana
ger.java:353)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
at java.lang.Thread.run(Thread.java:619)
if I did cassandra repair on all the servers on the cluster can I simply
truncate the column family HintsColumnFamily?
Will I lose any data?
Thanks
Michael
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Monday, November 21, 2011 8:40 PM
To: user@cassandra.apache.org
Cc: cassandra-user@incubator.apache.org
Subject: Re: Upgrade Cassandra Cluster to 1.0.3
Sounds to me like
https://issues.apache.org/jira/browse/CASSANDRA-3491, which was
present in 1.0.2 and fixed in 1.0.3. It sounds like you're running
the wrong version by mistake.
On Mon, Nov 21, 2011 at 7:53 AM, Michael Vaknine <mi...@citypath.com>
wrote:
> Hi,
> Any help will be appreciated.
>
> I am upgrading Cassandra 1.0.0 to 1.0.3 got error
> ERROR [CompactionExecutor:3] 2011-11-21 11:10:59,075
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor: 3,1,main]
> 3645 java.lang.StackOverflowError
> 3646 at com.google.common.base.Objects.equal(Objects.java:51)
> 3647 at org.apache.cassandra.utils.Pair.equals(Pair.java:48)
> 3648 at
>
java.util.concurrent.ConcurrentHashMap$Segment.get(ConcurrentHashMap.java:33
> 8)
> 3649 at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:769)
> 3650 at
>
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.get(Concurren
> tLinkedHashMap.java:740)
> 3651 at
>
org.apache.cassandra.cache.ConcurrentLinkedHashCache.get(ConcurrentLinkedHas
> hCache.java:81)
> 3652 at
>
org.apache.cassandra.cache.InstrumentingCache.get(InstrumentingCache.java:68
> )
> 3653 at
>
org.apache.cassandra.io.sstable.SSTableReader.getCachedPosition(SSTableReade
> r.java:598)
> 3654 at
>
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java
> :621)
> 3655 at
>
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader
> .java:786)
> 3656 at
>
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNa
> mesIterator.java:61)
>
> I tried to increase Xss to 640M and then got this error
>
> ERROR [Thread-28] 2011-11-21 12:52:40,808 AbstractCassandraDaemon.java
(line
> 133) Fatal exception in thread Thread[Thread-28,5,main]
> 4940 java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.Error: Maximum lock count exceeded
> 4941 at
>
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
> es(SecondaryIndexManager.java:131)
> 4942 at
>
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSessi
> on.java:151)
> 4943 at
>
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReade
> r.java:102)
> 4944 at
>
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.
> java:184)
> 4945 at
>
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
> a:81)
> 4946 Caused by: java.util.concurrent.ExecutionException: java.lang.Error:
> Maximum lock count exceeded
> 4947 at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> 4948 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> 4949 at
>
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
> es(SecondaryIndexManager.java:122)
> 4950 ... 4 more
> 4951 Caused by: java.lang.Error: Maximum lock count exceeded
>
> None of this errors happen on 1.0.0
>
>
> Thanks
> Michael
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Upgrade Cassandra Cluster to 1.0.3
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds to me like
https://issues.apache.org/jira/browse/CASSANDRA-3491, which was
present in 1.0.2 and fixed in 1.0.3. It sounds like you're running
the wrong version by mistake.
On Mon, Nov 21, 2011 at 7:53 AM, Michael Vaknine <mi...@citypath.com> wrote:
> Hi,
> Any help will be appreciated.
>
> I am upgrading Cassandra 1.0.0 to 1.0.3 got error
> ERROR [CompactionExecutor:3] 2011-11-21 11:10:59,075
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor: 3,1,main]
> 3645 java.lang.StackOverflowError
> 3646 at com.google.common.base.Objects.equal(Objects.java:51)
> 3647 at org.apache.cassandra.utils.Pair.equals(Pair.java:48)
> 3648 at
> java.util.concurrent.ConcurrentHashMap$Segment.get(ConcurrentHashMap.java:33
> 8)
> 3649 at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:769)
> 3650 at
> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.get(Concurren
> tLinkedHashMap.java:740)
> 3651 at
> org.apache.cassandra.cache.ConcurrentLinkedHashCache.get(ConcurrentLinkedHas
> hCache.java:81)
> 3652 at
> org.apache.cassandra.cache.InstrumentingCache.get(InstrumentingCache.java:68
> )
> 3653 at
> org.apache.cassandra.io.sstable.SSTableReader.getCachedPosition(SSTableReade
> r.java:598)
> 3654 at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java
> :621)
> 3655 at
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader
> .java:786)
> 3656 at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNa
> mesIterator.java:61)
>
> I tried to increase Xss to 640M and then got this error
>
> ERROR [Thread-28] 2011-11-21 12:52:40,808 AbstractCassandraDaemon.java (line
> 133) Fatal exception in thread Thread[Thread-28,5,main]
> 4940 java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.Error: Maximum lock count exceeded
> 4941 at
> org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
> es(SecondaryIndexManager.java:131)
> 4942 at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSessi
> on.java:151)
> 4943 at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReade
> r.java:102)
> 4944 at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.
> java:184)
> 4945 at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
> a:81)
> 4946 Caused by: java.util.concurrent.ExecutionException: java.lang.Error:
> Maximum lock count exceeded
> 4947 at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> 4948 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> 4949 at
> org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
> es(SecondaryIndexManager.java:122)
> 4950 ... 4 more
> 4951 Caused by: java.lang.Error: Maximum lock count exceeded
>
> None of this errors happen on 1.0.0
>
>
> Thanks
> Michael
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Upgrade Cassandra Cluster to 1.0.3
Posted by Michael Vaknine <mi...@citypath.com>.
Hi,
Any help will be appreciated.
I am upgrading Cassandra 1.0.0 to 1.0.3 got error
ERROR [CompactionExecutor:3] 2011-11-21 11:10:59,075
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor: 3,1,main]
3645 java.lang.StackOverflowError
3646 at com.google.common.base.Objects.equal(Objects.java:51)
3647 at org.apache.cassandra.utils.Pair.equals(Pair.java:48)
3648 at
java.util.concurrent.ConcurrentHashMap$Segment.get(ConcurrentHashMap.java:33
8)
3649 at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:769)
3650 at
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.get(Concurren
tLinkedHashMap.java:740)
3651 at
org.apache.cassandra.cache.ConcurrentLinkedHashCache.get(ConcurrentLinkedHas
hCache.java:81)
3652 at
org.apache.cassandra.cache.InstrumentingCache.get(InstrumentingCache.java:68
)
3653 at
org.apache.cassandra.io.sstable.SSTableReader.getCachedPosition(SSTableReade
r.java:598)
3654 at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java
:621)
3655 at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader
.java:786)
3656 at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNa
mesIterator.java:61)
I tried to increase Xss to 640M and then got this error
ERROR [Thread-28] 2011-11-21 12:52:40,808 AbstractCassandraDaemon.java (line
133) Fatal exception in thread Thread[Thread-28,5,main]
4940 java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.Error: Maximum lock count exceeded
4941 at
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
es(SecondaryIndexManager.java:131)
4942 at
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSessi
on.java:151)
4943 at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReade
r.java:102)
4944 at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.
java:184)
4945 at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
a:81)
4946 Caused by: java.util.concurrent.ExecutionException: java.lang.Error:
Maximum lock count exceeded
4947 at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
4948 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
4949 at
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndex
es(SecondaryIndexManager.java:122)
4950 ... 4 more
4951 Caused by: java.lang.Error: Maximum lock count exceeded
None of this errors happen on 1.0.0
Thanks
Michael
RE: Upgrade Cassandra Cluster to 1.0.2
Posted by Michael Vaknine <mi...@citypath.com>.
I am running java version:
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
I have tried to run with -Xss96k or -Xss256k or -Xss320 but it still give me
the error
TheNode has 8GB memory and %GB allocated to JAVA
ERROR [MutationStage:104] 2011-11-15 08:55:59,408
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[MutationStage:104,5,main]
java.lang.StackOverflowError
at
org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:209)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation
Controller.java:74)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont
roller.java:62)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto
re.java:1278)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1164)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1126)
at
org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507)
at org.apache.cassandra.db.Table.apply(Table.java:444)
at
org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:260)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation
Controller.java:159)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont
roller.java:62)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto
re.java:1278)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1164)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1126)
at
org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507)
at org.apache.cassandra.db.Table.apply(Table.java:444)
at
org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:260)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation
Controller.java:159)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont
roller.java:62)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto
re.java:1278)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1164)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1126)
at
org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507)
at org.apache.cassandra.db.Table.apply(Table.java:444)
I ma getting WARN as well after changing JAVA
WARN [MutationStage:44] 2011-11-15 08:51:57,479 Memtable.java (line 142)
MemoryMeter uninitialized (jamm not specified as java agent); assuming
liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm
because you are using a buggy JRE; upgrade to the Sun JRE instead
-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com]
Sent: Monday, November 14, 2011 7:53 PM
To: user@cassandra.apache.org
Cc: cassandra-user@incubator.apache.org
Subject: Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 8:06 AM, Michael Vaknine <mi...@citypath.com>
wrote:
> Well,
> I tried to delete the hints on the failed cluster but I could not start it
> I got other errors such as
>
> ERROR [MutationStage:34] 2011-11-14 15:37:43,813
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[MutationStage:34,5,main]
> java.lang.StackOverflowError
> at
>
java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField
> UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216)
This is different an unrelated to hints. What jvm are you using? It
looks like it just simply ran out of stack space, which is odd, but
you can control that with the -Xss option if needed.
-Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Nov 14, 2011 at 8:06 AM, Michael Vaknine <mi...@citypath.com> wrote:
> Well,
> I tried to delete the hints on the failed cluster but I could not start it
> I got other errors such as
>
> ERROR [MutationStage:34] 2011-11-14 15:37:43,813
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[MutationStage:34,5,main]
> java.lang.StackOverflowError
> at
> java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField
> UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216)
This is different an unrelated to hints. What jvm are you using? It
looks like it just simply ran out of stack space, which is odd, but
you can control that with the -Xss option if needed.
-Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
Posted by Michael Vaknine <mi...@citypath.com>.
Well,
I tried to delete the hints on the failed cluster but I could not start it
I got other errors such as
ERROR [MutationStage:34] 2011-11-14 15:37:43,813
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[MutationStage:34,5,main]
java.lang.StackOverflowError
at
java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField
UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216)
at
java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField
UpdaterImpl.compareAndSet(AtomicReferenceFieldUpdater.java:227)
at
java.util.concurrent.ConcurrentSkipListMap.casHead(ConcurrentSkipListMap.jav
a:359)
at
java.util.concurrent.ConcurrentSkipListMap.insertIndex(ConcurrentSkipListMap
.java:968)
at
java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:
898)
at
java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap
.java:1893)
at
org.apache.cassandra.db.ThreadSafeSortedColumns.addColumn(ThreadSafeSortedCo
lumns.java:88)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCont
ainer.java:122)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCont
ainer.java:117)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColum
ns(SSTableNamesIterator.java:183)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableName
sIterator.java:130)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNa
mesIterator.java:72)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(Nam
esQueryFilter.java:60)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFil
ter.java:78)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation
Controller.java:113)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont
roller.java:62)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto
re.java:1278)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1164)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.
java:1126)
at
org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507)
187179,2-9 95%
Should I have deleted all the hints on all the clustr?
Is 1.0.0 stable enough? I am checking this and don't want to upgrade again
in a week or so.
Thanks
Michael
-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com]
Sent: Monday, November 14, 2011 4:01 PM
To: user@cassandra.apache.org
Cc: cassandra-user@incubator.apache.org
Subject: Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 7:53 AM, Michael Vaknine <mi...@citypath.com>
wrote:
> Does this means that I have to wait to 1.0.3?
In the meantime you can just delete the hints and rely on read repair
or antientropy repair if you're concerned about the consistency of
your replicas.
-Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Nov 14, 2011 at 7:53 AM, Michael Vaknine <mi...@citypath.com> wrote:
> Does this means that I have to wait to 1.0.3?
In the meantime you can just delete the hints and rely on read repair
or antientropy repair if you're concerned about the consistency of
your replicas.
-Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
Posted by Michael Vaknine <mi...@citypath.com>.
Does this means that I have to wait to 1.0.3?
-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com]
Sent: Monday, November 14, 2011 3:51 PM
To: user@cassandra.apache.org
Cc: cassandra-user@incubator.apache.org
Subject: Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 1:21 AM, Michael Vaknine <mi...@citypath.com>
wrote:
> Hi,
>
> After configuring the encryption on Cassandra.yaml I get this error when
> upgrading from 1.0.0 to 1.0.2
> Attached the log file with the errors.
https://issues.apache.org/jira/browse/CASSANDRA-3466
-Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Nov 14, 2011 at 1:21 AM, Michael Vaknine <mi...@citypath.com> wrote:
> Hi,
>
> After configuring the encryption on Cassandra.yaml I get this error when
> upgrading from 1.0.0 to 1.0.2
> Attached the log file with the errors.
https://issues.apache.org/jira/browse/CASSANDRA-3466
-Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
Posted by Michael Vaknine <mi...@citypath.com>.
Hi,
After configuring the encryption on Cassandra.yaml I get this error when
upgrading from 1.0.0 to 1.0.2
Attached the log file with the errors.
Thanks
Michael Vaknine
-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com]
Sent: Sunday, November 13, 2011 6:33 PM
To: user@cassandra.apache.org
Cc: cassandra-user@incubator.apache.org
Subject: Re: Upgrade Cassandra Cluster to 1.0.2
I believe https://issues.apache.org/jira/browse/CASSANDRA-2802 broke
it. I've created https://issues.apache.org/jira/browse/CASSANDRA-3489
to address this separately.
On Sun, Nov 13, 2011 at 9:37 AM, Michael Vaknine <mi...@citypath.com>
wrote:
> You are right this solved the problem.
> I do not understand why version 1.0.0 was not affected since I used the
same
> configuration yaml file.
>
> Thank you.
> Michael Vaknine
>
>
> -----Original Message-----
> From: Brandon Williams [mailto:driftx@gmail.com]
> Sent: Sunday, November 13, 2011 4:48 PM
> To: user@cassandra.apache.org
> Cc: cassandra-user@incubator.apache.org
> Subject: Re: Upgrade Cassandra Cluster to 1.0.2
>
> On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine <mi...@citypath.com>
> wrote:
>> I am trying to upgrade to 1.0.2 and when I try to start the first
upgraded
>> server I get the following error
>>
>>
>>
>> ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[WRITE-/10.5.6.102,5,main]
>>
>> java.lang.NullPointerException
>>
>> at
>>
>
org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou
> ndTcpConnectionPool.java:93)
>>
>> at
>>
>
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn
> ectionPool.java:77)
>>
>> at
>>
>
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection
> .java:209)
>>
>> at
>>
>
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
> a:93)
>
> You are probably missing the encryption options in your yaml. I
> noticed this problem as part of
> https://issues.apache.org/jira/browse/CASSANDRA-3045.
>
> -Brandon
>
>
Re: Upgrade Cassandra Cluster to 1.0.2
Posted by Brandon Williams <dr...@gmail.com>.
I believe https://issues.apache.org/jira/browse/CASSANDRA-2802 broke
it. I've created https://issues.apache.org/jira/browse/CASSANDRA-3489
to address this separately.
On Sun, Nov 13, 2011 at 9:37 AM, Michael Vaknine <mi...@citypath.com> wrote:
> You are right this solved the problem.
> I do not understand why version 1.0.0 was not affected since I used the same
> configuration yaml file.
>
> Thank you.
> Michael Vaknine
>
>
> -----Original Message-----
> From: Brandon Williams [mailto:driftx@gmail.com]
> Sent: Sunday, November 13, 2011 4:48 PM
> To: user@cassandra.apache.org
> Cc: cassandra-user@incubator.apache.org
> Subject: Re: Upgrade Cassandra Cluster to 1.0.2
>
> On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine <mi...@citypath.com>
> wrote:
>> I am trying to upgrade to 1.0.2 and when I try to start the first upgraded
>> server I get the following error
>>
>>
>>
>> ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[WRITE-/10.5.6.102,5,main]
>>
>> java.lang.NullPointerException
>>
>> at
>>
> org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou
> ndTcpConnectionPool.java:93)
>>
>> at
>>
> org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn
> ectionPool.java:77)
>>
>> at
>>
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection
> .java:209)
>>
>> at
>>
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
> a:93)
>
> You are probably missing the encryption options in your yaml. I
> noticed this problem as part of
> https://issues.apache.org/jira/browse/CASSANDRA-3045.
>
> -Brandon
>
>
RE: Upgrade Cassandra Cluster to 1.0.2
Posted by Michael Vaknine <mi...@citypath.com>.
You are right this solved the problem.
I do not understand why version 1.0.0 was not affected since I used the same
configuration yaml file.
Thank you.
Michael Vaknine
-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com]
Sent: Sunday, November 13, 2011 4:48 PM
To: user@cassandra.apache.org
Cc: cassandra-user@incubator.apache.org
Subject: Re: Upgrade Cassandra Cluster to 1.0.2
On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine <mi...@citypath.com>
wrote:
> I am trying to upgrade to 1.0.2 and when I try to start the first upgraded
> server I get the following error
>
>
>
> ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[WRITE-/10.5.6.102,5,main]
>
> java.lang.NullPointerException
>
> at
>
org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou
ndTcpConnectionPool.java:93)
>
> at
>
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn
ectionPool.java:77)
>
> at
>
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection
.java:209)
>
> at
>
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:93)
You are probably missing the encryption options in your yaml. I
noticed this problem as part of
https://issues.apache.org/jira/browse/CASSANDRA-3045.
-Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
Posted by Brandon Williams <dr...@gmail.com>.
On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine <mi...@citypath.com> wrote:
> I am trying to upgrade to 1.0.2 and when I try to start the first upgraded
> server I get the following error
>
>
>
> ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[WRITE-/10.5.6.102,5,main]
>
> java.lang.NullPointerException
>
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(OutboundTcpConnectionPool.java:93)
>
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:77)
>
> at
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:209)
>
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:93)
You are probably missing the encryption options in your yaml. I
noticed this problem as part of
https://issues.apache.org/jira/browse/CASSANDRA-3045.
-Brandon