You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Boris Yen <yu...@gmail.com> on 2011/07/20 13:25:46 UTC
What is the nodeId for?
Hi,
I think we might have screwed our data up. I saw multiple columns inside
record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
I was wondering if anyone could tell me what the NodeId is for? so that I
might be able to duplicate this.
Thanks in advance
Boris
Re: What is the nodeId for?
Posted by Sylvain Lebresne <sy...@datastax.com>.
That looks very much the same, yes.
--
Sylvain
On Thu, Jul 21, 2011 at 3:15 AM, Boris Yen <yu...@gmail.com> wrote:
> Not sure if this is the same. I saw exceptions like this:
> INFO 15:33:49,336 Finished reading
> /root/commitlog_tmp/CommitLog-1311135088656.log
> ERROR 15:33:49,336 Exception encountered during startup.
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.AssertionError
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:509)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:332)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:174)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:505)
> ... 5 more
> Caused by: java.lang.AssertionError
> at
> org.apache.cassandra.db.SystemTable.getCurrentLocalNodeId(SystemTable.java:383)
> at
> org.apache.cassandra.utils.NodeId$LocalNodeIdHistory.<init>(NodeId.java:179)
> at org.apache.cassandra.utils.NodeId.<clinit>(NodeId.java:38)
> at
> org.apache.cassandra.db.context.CounterContext$ContextState.getNodeId(CounterContext.java:722)
> at
> org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:333)
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
> at
> org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:185)
> at
> org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:204)
> at
> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:228)
> at
> org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
> at
> org.apache.cassandra.db.ColumnFamily.resolve(ColumnFamily.java:414)
> at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:148)
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:774)
> at org.apache.cassandra.db.Table.apply(Table.java:451)
> at
> org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:324)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> On Wed, Jul 20, 2011 at 11:42 PM, Sylvain Lebresne <sy...@datastax.com>
> wrote:
>>
>> Possibly, you've hitted this:
>> https://issues.apache.org/jira/browse/CASSANDRA-2824
>> Should be fixed in next minor release.
>>
>> In the meantime, you "fix" should be alright.
>>
>> --
>> Sylvain
>>
>>
>> On Wed, Jul 20, 2011 at 3:47 PM, Boris Yen <yu...@gmail.com> wrote:
>> > Hi Sam,
>> > Thanks for the explanation.
>> > The NodeIds do appear in the Local row of NodeIdInfo, and after manually
>> > deleting two (I got three before I deleted them) of them from
>> > CurrentLocal
>> > row, the cassandra can be restarted now. I was just thinking what could
>> > be
>> > the possible cause for this? and wondering if anyone has any idea about
>> > this?
>> > Boris
>> >
>> > On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton <so...@acunu.com> wrote:
>> >>
>> >> The NodeId is used in counter replication. Counters are stored on each
>> >> replica as a set of "shards," where each shard corresponds to the local
>> >> count of one of the replicas for that counter, as identified by the
>> >> NodeId.
>> >> A NodeId is generated the first time cassandra starts, and might be
>> >> renewed during cleanup, or if you explicitly tell cassandra to generate
>> >> a
>> >> new one by starting it with -Dcassandra.renew_counter_id
>> >> Do either of the NodeIds in the CurrentLocal row also appear in the
>> >> Local
>> >> row of the NodeIdInfo CF? the Local row is a history of previous
>> >> NodeIds
>> >> belonging to this node. If one appears in the history and the other
>> >> does
>> >> not, it is probably safe to assume that the old NodeId was not deleted
>> >> from
>> >> the CurrentLocal row correctly, and so you could try removing it from
>> >> there
>> >> manually.
>> >>
>> >> Sam
>> >> --
>> >> Sam Overton
>> >> Acunu | http://www.acunu.com | @acunu
>> >> On 20 July 2011 12:25, Boris Yen <yu...@gmail.com> wrote:
>> >>>
>> >>> Hi,
>> >>> I think we might have screwed our data up. I saw multiple columns
>> >>> inside
>> >>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead
>> >>> forever.
>> >>> I was wondering if anyone could tell me what the NodeId is for? so
>> >>> that I
>> >>> might be able to duplicate this.
>> >>> Thanks in advance
>> >>> Boris
>> >
>> >
>
>
Re: What is the nodeId for?
Posted by Boris Yen <yu...@gmail.com>.
Not sure if this is the same. I saw exceptions like this:
INFO 15:33:49,336 Finished reading
/root/commitlog_tmp/CommitLog-1311135088656.log
ERROR 15:33:49,336 Exception encountered during startup.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:509)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:332)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:174)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:505)
... 5 more
Caused by: java.lang.AssertionError
at
org.apache.cassandra.db.SystemTable.getCurrentLocalNodeId(SystemTable.java:383)
at
org.apache.cassandra.utils.NodeId$LocalNodeIdHistory.<init>(NodeId.java:179)
at org.apache.cassandra.utils.NodeId.<clinit>(NodeId.java:38)
at
org.apache.cassandra.db.context.CounterContext$ContextState.getNodeId(CounterContext.java:722)
at
org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:333)
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
at
org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:185)
at
org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:204)
at
org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:228)
at
org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
at
org.apache.cassandra.db.ColumnFamily.resolve(ColumnFamily.java:414)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
at org.apache.cassandra.db.Memtable.put(Memtable.java:148)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:774)
at org.apache.cassandra.db.Table.apply(Table.java:451)
at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:324)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
On Wed, Jul 20, 2011 at 11:42 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> Possibly, you've hitted this:
> https://issues.apache.org/jira/browse/CASSANDRA-2824
> Should be fixed in next minor release.
>
> In the meantime, you "fix" should be alright.
>
> --
> Sylvain
>
>
> On Wed, Jul 20, 2011 at 3:47 PM, Boris Yen <yu...@gmail.com> wrote:
> > Hi Sam,
> > Thanks for the explanation.
> > The NodeIds do appear in the Local row of NodeIdInfo, and after manually
> > deleting two (I got three before I deleted them) of them from
> CurrentLocal
> > row, the cassandra can be restarted now. I was just thinking what could
> be
> > the possible cause for this? and wondering if anyone has any idea about
> > this?
> > Boris
> >
> > On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton <so...@acunu.com> wrote:
> >>
> >> The NodeId is used in counter replication. Counters are stored on each
> >> replica as a set of "shards," where each shard corresponds to the local
> >> count of one of the replicas for that counter, as identified by the
> NodeId.
> >> A NodeId is generated the first time cassandra starts, and might be
> >> renewed during cleanup, or if you explicitly tell cassandra to generate
> a
> >> new one by starting it with -Dcassandra.renew_counter_id
> >> Do either of the NodeIds in the CurrentLocal row also appear in the
> Local
> >> row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
> >> belonging to this node. If one appears in the history and the other does
> >> not, it is probably safe to assume that the old NodeId was not deleted
> from
> >> the CurrentLocal row correctly, and so you could try removing it from
> there
> >> manually.
> >>
> >> Sam
> >> --
> >> Sam Overton
> >> Acunu | http://www.acunu.com | @acunu
> >> On 20 July 2011 12:25, Boris Yen <yu...@gmail.com> wrote:
> >>>
> >>> Hi,
> >>> I think we might have screwed our data up. I saw multiple columns
> inside
> >>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead
> forever.
> >>> I was wondering if anyone could tell me what the NodeId is for? so that
> I
> >>> might be able to duplicate this.
> >>> Thanks in advance
> >>> Boris
> >
> >
>
Re: What is the nodeId for?
Posted by Sylvain Lebresne <sy...@datastax.com>.
Possibly, you've hitted this:
https://issues.apache.org/jira/browse/CASSANDRA-2824
Should be fixed in next minor release.
In the meantime, you "fix" should be alright.
--
Sylvain
On Wed, Jul 20, 2011 at 3:47 PM, Boris Yen <yu...@gmail.com> wrote:
> Hi Sam,
> Thanks for the explanation.
> The NodeIds do appear in the Local row of NodeIdInfo, and after manually
> deleting two (I got three before I deleted them) of them from CurrentLocal
> row, the cassandra can be restarted now. I was just thinking what could be
> the possible cause for this? and wondering if anyone has any idea about
> this?
> Boris
>
> On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton <so...@acunu.com> wrote:
>>
>> The NodeId is used in counter replication. Counters are stored on each
>> replica as a set of "shards," where each shard corresponds to the local
>> count of one of the replicas for that counter, as identified by the NodeId.
>> A NodeId is generated the first time cassandra starts, and might be
>> renewed during cleanup, or if you explicitly tell cassandra to generate a
>> new one by starting it with -Dcassandra.renew_counter_id
>> Do either of the NodeIds in the CurrentLocal row also appear in the Local
>> row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
>> belonging to this node. If one appears in the history and the other does
>> not, it is probably safe to assume that the old NodeId was not deleted from
>> the CurrentLocal row correctly, and so you could try removing it from there
>> manually.
>>
>> Sam
>> --
>> Sam Overton
>> Acunu | http://www.acunu.com | @acunu
>> On 20 July 2011 12:25, Boris Yen <yu...@gmail.com> wrote:
>>>
>>> Hi,
>>> I think we might have screwed our data up. I saw multiple columns inside
>>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
>>> I was wondering if anyone could tell me what the NodeId is for? so that I
>>> might be able to duplicate this.
>>> Thanks in advance
>>> Boris
>
>
Re: What is the nodeId for?
Posted by Boris Yen <yu...@gmail.com>.
Hi Sam,
Thanks for the explanation.
The NodeIds do appear in the Local row of NodeIdInfo, and after manually
deleting two (I got three before I deleted them) of them from CurrentLocal
row, the cassandra can be restarted now. I was just thinking what could be
the possible cause for this? and wondering if anyone has any idea about
this?
Boris
On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton <so...@acunu.com> wrote:
> The NodeId is used in counter replication. Counters are stored on each
> replica as a set of "shards," where each shard corresponds to the local
> count of one of the replicas for that counter, as identified by the NodeId.
>
> A NodeId is generated the first time cassandra starts, and might be renewed
> during cleanup, or if you explicitly tell cassandra to generate a new one by
> starting it with -D*cassandra*.renew_counter_id
>
> Do either of the NodeIds in the CurrentLocal row also appear in the Local
> row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
> belonging to this node. If one appears in the history and the other does
> not, it is probably safe to assume that the old NodeId was not deleted from
> the CurrentLocal row correctly, and so you could try removing it from there
> manually.
>
> Sam
> --
> Sam Overton
> Acunu | http://www.acunu.com | @acunu
>
> On 20 July 2011 12:25, Boris Yen <yu...@gmail.com> wrote:
>
>> Hi,
>>
>> I think we might have screwed our data up. I saw multiple columns inside
>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
>> I was wondering if anyone could tell me what the NodeId is for? so that I
>> might be able to duplicate this.
>>
>> Thanks in advance
>> Boris
>>
>
>
Re: What is the nodeId for?
Posted by Sam Overton <so...@acunu.com>.
The NodeId is used in counter replication. Counters are stored on each
replica as a set of "shards," where each shard corresponds to the local
count of one of the replicas for that counter, as identified by the NodeId.
A NodeId is generated the first time cassandra starts, and might be renewed
during cleanup, or if you explicitly tell cassandra to generate a new one by
starting it with -D*cassandra*.renew_counter_id
Do either of the NodeIds in the CurrentLocal row also appear in the Local
row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
belonging to this node. If one appears in the history and the other does
not, it is probably safe to assume that the old NodeId was not deleted from
the CurrentLocal row correctly, and so you could try removing it from there
manually.
Sam
--
Sam Overton
Acunu | http://www.acunu.com | @acunu
On 20 July 2011 12:25, Boris Yen <yu...@gmail.com> wrote:
> Hi,
>
> I think we might have screwed our data up. I saw multiple columns inside
> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
> I was wondering if anyone could tell me what the NodeId is for? so that I
> might be able to duplicate this.
>
> Thanks in advance
> Boris
>