You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Alain RODRIGUEZ <ar...@gmail.com> on 2012/11/15 15:21:27 UTC
Upgrade 1.1.2 -> 1.1.6
We had an issue with counters over-counting even using the nodetool drain
command before upgrading...
Here is my bash history
69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
70 cp /etc/cassandra/cassandra-env.sh
/etc/cassandra/cassandra-env.sh.bak
71 sudo apt-get install cassandra
72 nodetool disablethrift
73 nodetool drain
74 service cassandra stop
75 cat /etc/cassandra/cassandra-env.sh
/etc/cassandra/cassandra-env.sh.bak
76 vim /etc/cassandra/cassandra-env.sh
77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
78 vim /etc/cassandra/cassandra.yaml
79 service cassandra start
So I think I followed these steps
http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
I merged my conf files with an external tool so consider I merged my conf
files on steps 76 and 78.
I saw that the "sudo apt-get install cassandra" stop the server and restart
it automatically. So it updated without draining and restart before I had
the time to reconfigure the conf files. Is this "normal" ? Is there a way
to avoid it ?
So for the second node I decided to try to stop C*before the upgrade.
125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
126 cp /etc/cassandra/cassandra-env.sh
/etc/cassandra/cassandra-env.sh.bak
127 nodetool disablegossip
128 nodetool disablethrift
129 nodetool drain
130 service cassandra stop
131 sudo apt-get install cassandra
//131 : This restarted cassandra
132 nodetool disablethrift
133 nodetool disablegossip
134 nodetool drain
135 service cassandra stop
136 cat /etc/cassandra/cassandra-env.sh
/etc/cassandra/cassandra-env.sh.bak
137 cim /etc/cassandra/cassandra-env.sh
138 vim /etc/cassandra/cassandra-env.sh
139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
140 vim /etc/cassandra/cassandra.yaml
141 service cassandra start
After both of these updates I saw my current counters increase without any
reason.
Did I do anything wrong ?
Alain
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Mike Heffner <mi...@librato.com>.
Alain,
My understanding is that drain ensures that all memtables are flushed, so
that there is no data in the commitlog that is isn't in an sstable. A
marker is saved that indicates the commit logs should not be replayed.
Commitlogs are only removed from disk periodically
(after commitlog_total_space_in_mb is exceeded?).
With 1.1.5/6, all nanotime commitlogs are replayed on startup regardless of
whether they've been flushed. So in our case manually removing all the
commitlogs after a drain was the only way to prevent their replay.
Mike
On Tue, Nov 20, 2012 at 5:19 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> @Mike
>
> I am glad to see I am not the only one with this issue (even if I am sorry
> it happened to you of course.).
>
> Isn't drain supposed to clear the commit logs ? Did removing them worked
> properly ?
>
> I his warning to C* users, Jonathan Ellis told that a drain would avoid
> this issue, It seems like it doesn't.
>
> @Rob
>
> You understood precisely the 2 issues I met during the upgrade. I am sad
> to see none of them is yet resolved and probably wont.
>
>
> 2012/11/20 Mike Heffner <mi...@librato.com>
>
>> Alain,
>>
>> We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs
>> replayed regardless of the drain. After noticing this on the first node, we
>> did the following:
>>
>> * nodetool flush
>> * nodetool drain
>> * service cassandra stop
>> * mv /path/to/logs/*.log /backup/
>> * apt-get install cassandra
>> <restarts automatically>
>>
>> I also agree that starting C* after an upgrade/install seems quite broken
>> if it was already stopped before the install. However annoying, I have
>> found this to be the default for most Ubuntu daemon packages.
>>
>> Mike
>>
>>
>> On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>>
>>> We had an issue with counters over-counting even using the nodetool
>>> drain command before upgrading...
>>>
>>> Here is my bash history
>>>
>>> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>> 70 cp /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 71 sudo apt-get install cassandra
>>> 72 nodetool disablethrift
>>> 73 nodetool drain
>>> 74 service cassandra stop
>>> 75 cat /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 76 vim /etc/cassandra/cassandra-env.sh
>>> 77 cat /etc/cassandra/cassandra.yaml
>>> /etc/cassandra/cassandra.yaml.bak
>>> 78 vim /etc/cassandra/cassandra.yaml
>>> 79 service cassandra start
>>>
>>> So I think I followed these steps
>>> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>>>
>>> I merged my conf files with an external tool so consider I merged my
>>> conf files on steps 76 and 78.
>>>
>>> I saw that the "sudo apt-get install cassandra" stop the server and
>>> restart it automatically. So it updated without draining and restart before
>>> I had the time to reconfigure the conf files. Is this "normal" ? Is there a
>>> way to avoid it ?
>>>
>>> So for the second node I decided to try to stop C*before the upgrade.
>>>
>>> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>> 126 cp /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 127 nodetool disablegossip
>>> 128 nodetool disablethrift
>>> 129 nodetool drain
>>> 130 service cassandra stop
>>> 131 sudo apt-get install cassandra
>>>
>>> //131 : This restarted cassandra
>>>
>>> 132 nodetool disablethrift
>>> 133 nodetool disablegossip
>>> 134 nodetool drain
>>> 135 service cassandra stop
>>> 136 cat /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 137 cim /etc/cassandra/cassandra-env.sh
>>> 138 vim /etc/cassandra/cassandra-env.sh
>>> 139 cat /etc/cassandra/cassandra.yaml
>>> /etc/cassandra/cassandra.yaml.bak
>>> 140 vim /etc/cassandra/cassandra.yaml
>>> 141 service cassandra start
>>>
>>> After both of these updates I saw my current counters increase without
>>> any reason.
>>>
>>> Did I do anything wrong ?
>>>
>>> Alain
>>>
>>>
>>
>>
>> --
>>
>> Mike Heffner <mi...@librato.com>
>> Librato, Inc.
>>
>>
>>
>
--
Mike Heffner <mi...@librato.com>
Librato, Inc.
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
@Mike
I am glad to see I am not the only one with this issue (even if I am sorry
it happened to you of course.).
Isn't drain supposed to clear the commit logs ? Did removing them worked
properly ?
I his warning to C* users, Jonathan Ellis told that a drain would avoid
this issue, It seems like it doesn't.
@Rob
You understood precisely the 2 issues I met during the upgrade. I am sad to
see none of them is yet resolved and probably wont.
2012/11/20 Mike Heffner <mi...@librato.com>
> Alain,
>
> We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs replayed
> regardless of the drain. After noticing this on the first node, we did the
> following:
>
> * nodetool flush
> * nodetool drain
> * service cassandra stop
> * mv /path/to/logs/*.log /backup/
> * apt-get install cassandra
> <restarts automatically>
>
> I also agree that starting C* after an upgrade/install seems quite broken
> if it was already stopped before the install. However annoying, I have
> found this to be the default for most Ubuntu daemon packages.
>
> Mike
>
>
> On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>
>> We had an issue with counters over-counting even using the nodetool drain
>> command before upgrading...
>>
>> Here is my bash history
>>
>> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 70 cp /etc/cassandra/cassandra-env.sh
>> /etc/cassandra/cassandra-env.sh.bak
>> 71 sudo apt-get install cassandra
>> 72 nodetool disablethrift
>> 73 nodetool drain
>> 74 service cassandra stop
>> 75 cat /etc/cassandra/cassandra-env.sh
>> /etc/cassandra/cassandra-env.sh.bak
>> 76 vim /etc/cassandra/cassandra-env.sh
>> 77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 78 vim /etc/cassandra/cassandra.yaml
>> 79 service cassandra start
>>
>> So I think I followed these steps
>> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>>
>> I merged my conf files with an external tool so consider I merged my conf
>> files on steps 76 and 78.
>>
>> I saw that the "sudo apt-get install cassandra" stop the server and
>> restart it automatically. So it updated without draining and restart before
>> I had the time to reconfigure the conf files. Is this "normal" ? Is there a
>> way to avoid it ?
>>
>> So for the second node I decided to try to stop C*before the upgrade.
>>
>> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 126 cp /etc/cassandra/cassandra-env.sh
>> /etc/cassandra/cassandra-env.sh.bak
>> 127 nodetool disablegossip
>> 128 nodetool disablethrift
>> 129 nodetool drain
>> 130 service cassandra stop
>> 131 sudo apt-get install cassandra
>>
>> //131 : This restarted cassandra
>>
>> 132 nodetool disablethrift
>> 133 nodetool disablegossip
>> 134 nodetool drain
>> 135 service cassandra stop
>> 136 cat /etc/cassandra/cassandra-env.sh
>> /etc/cassandra/cassandra-env.sh.bak
>> 137 cim /etc/cassandra/cassandra-env.sh
>> 138 vim /etc/cassandra/cassandra-env.sh
>> 139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 140 vim /etc/cassandra/cassandra.yaml
>> 141 service cassandra start
>>
>> After both of these updates I saw my current counters increase without
>> any reason.
>>
>> Did I do anything wrong ?
>>
>> Alain
>>
>>
>
>
> --
>
> Mike Heffner <mi...@librato.com>
> Librato, Inc.
>
>
>
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Tamar Fraenkel <ta...@tok-media.com>.
Hi!
I had the same problem (over counting due to replay of commit log, which
ignored drain) after upgrading my cluster from 1.0.9 to 1.0.11.
I updated the Cassandra tickets mentioned in this thread.
Regards,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Tue, Nov 20, 2012 at 11:03 PM, Mike Heffner <mi...@librato.com> wrote:
>
> On Tue, Nov 20, 2012 at 2:49 PM, Rob Coli <rc...@palominodb.com> wrote:
>
>> On Mon, Nov 19, 2012 at 7:18 PM, Mike Heffner <mi...@librato.com> wrote:
>> > We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs
>> replayed
>> > regardless of the drain.
>>
>> Your experience and desire for different (expected) behavior is welcomed
>> on :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-4446
>>
>> "nodetool drain sometimes doesn't mark commitlog fully flushed"
>>
>> If every production operator who experiences this issue shares their
>> experience on this bug, perhaps the project will acknowledge and
>> address it.
>>
>>
> Well in this case I think our issue was that upgrading from
> nanotime->epoch seconds, by definition, replays all commit logs. That's not
> due to any specific problem with nodetool drain not marking commitlog's
> flushed, but a safety to ensure data is not lost due to buggy nanotime
> implementations.
>
> For us, it was that the upgrade instructions pre-1.1.5->1.1.6 didn't
> mention that CL's should be removed if successfully drained. On the other
> hand, we do not use counters so replaying them was merely a much longer
> MTT-Return after restarting with 1.1.6.
>
> Mike
>
> --
>
> Mike Heffner <mi...@librato.com>
> Librato, Inc.
>
>
>
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Mike Heffner <mi...@librato.com>.
On Tue, Nov 20, 2012 at 2:49 PM, Rob Coli <rc...@palominodb.com> wrote:
> On Mon, Nov 19, 2012 at 7:18 PM, Mike Heffner <mi...@librato.com> wrote:
> > We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs
> replayed
> > regardless of the drain.
>
> Your experience and desire for different (expected) behavior is welcomed
> on :
>
> https://issues.apache.org/jira/browse/CASSANDRA-4446
>
> "nodetool drain sometimes doesn't mark commitlog fully flushed"
>
> If every production operator who experiences this issue shares their
> experience on this bug, perhaps the project will acknowledge and
> address it.
>
>
Well in this case I think our issue was that upgrading from nanotime->epoch
seconds, by definition, replays all commit logs. That's not due to any
specific problem with nodetool drain not marking commitlog's flushed, but a
safety to ensure data is not lost due to buggy nanotime implementations.
For us, it was that the upgrade instructions pre-1.1.5->1.1.6 didn't
mention that CL's should be removed if successfully drained. On the other
hand, we do not use counters so replaying them was merely a much longer
MTT-Return after restarting with 1.1.6.
Mike
--
Mike Heffner <mi...@librato.com>
Librato, Inc.
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Rob Coli <rc...@palominodb.com>.
On Mon, Nov 19, 2012 at 7:18 PM, Mike Heffner <mi...@librato.com> wrote:
> We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs replayed
> regardless of the drain.
Your experience and desire for different (expected) behavior is welcomed on :
https://issues.apache.org/jira/browse/CASSANDRA-4446
"nodetool drain sometimes doesn't mark commitlog fully flushed"
If every production operator who experiences this issue shares their
experience on this bug, perhaps the project will acknowledge and
address it.
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Mike Heffner <mi...@librato.com>.
Alain,
We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs replayed
regardless of the drain. After noticing this on the first node, we did the
following:
* nodetool flush
* nodetool drain
* service cassandra stop
* mv /path/to/logs/*.log /backup/
* apt-get install cassandra
<restarts automatically>
I also agree that starting C* after an upgrade/install seems quite broken
if it was already stopped before the install. However annoying, I have
found this to be the default for most Ubuntu daemon packages.
Mike
On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> We had an issue with counters over-counting even using the nodetool drain
> command before upgrading...
>
> Here is my bash history
>
> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 70 cp /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 71 sudo apt-get install cassandra
> 72 nodetool disablethrift
> 73 nodetool drain
> 74 service cassandra stop
> 75 cat /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 76 vim /etc/cassandra/cassandra-env.sh
> 77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 78 vim /etc/cassandra/cassandra.yaml
> 79 service cassandra start
>
> So I think I followed these steps
> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>
> I merged my conf files with an external tool so consider I merged my conf
> files on steps 76 and 78.
>
> I saw that the "sudo apt-get install cassandra" stop the server and
> restart it automatically. So it updated without draining and restart before
> I had the time to reconfigure the conf files. Is this "normal" ? Is there a
> way to avoid it ?
>
> So for the second node I decided to try to stop C*before the upgrade.
>
> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 126 cp /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 127 nodetool disablegossip
> 128 nodetool disablethrift
> 129 nodetool drain
> 130 service cassandra stop
> 131 sudo apt-get install cassandra
>
> //131 : This restarted cassandra
>
> 132 nodetool disablethrift
> 133 nodetool disablegossip
> 134 nodetool drain
> 135 service cassandra stop
> 136 cat /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 137 cim /etc/cassandra/cassandra-env.sh
> 138 vim /etc/cassandra/cassandra-env.sh
> 139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 140 vim /etc/cassandra/cassandra.yaml
> 141 service cassandra start
>
> After both of these updates I saw my current counters increase without any
> reason.
>
> Did I do anything wrong ?
>
> Alain
>
>
--
Mike Heffner <mi...@librato.com>
Librato, Inc.
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
"This looks like the counters were more out of sync before the upgrade than
after?"
My guess is the update makes some counters over-count since I saw the value
of the sum of our daily counter increase by 2000 after each restart at the
exact moment that the node is marked as being up. This counter was about
5000, after upgrading the first node 7000, after the second upgrade 9000
and finally 11000. This value shouldn't have increase since we had stopped
our storm topology and we were queuing events without writing directly into
C*.
"Do you know if your client is retrying counter operations ? (I saw some
dropped messages in the S1 log). "
I am using phpCassa written by Tyler Hobbs. I have a pool configured like
this:
$max_retries = 4
$send_timeout = 2000
$recv_timeout = 2000
$pool_size = NULL //max(5, count($servers) * 2)
$pool = new ConnectionPool($base, $servers, $pool_size, $max_retries,
$send_timeout, $recv_timeout);
So I guess my client is retrying count operations like any other operation,
4 times max with 2 sec time out.
"S1 shows a lot of Commit Log replay going on. Reading your timeline below
this sounds like the auto restart catching you out."
Is there a way to remove this auto-restart while upgrading from the
Datastax repository on Ubuntu ?
Let me know if you need something more to understand what happened.
Alain
Re: Upgrade 1.1.2 -> 1.1.6
Posted by aaron morton <aa...@thelastpickle.com>.
> time (UTC) 0 1 2 3 4 5 6 7 8 9 10 11 12 13
> Good value 88 44 26 35 26 86 187 251 455 389 473 367 453 373
> C* counter 149 82 45 68 38 146 329 414 746 566 473 377 453 373
> I finished my Cassandra 1.1.6 upgrades at 9:30 UTC.
This looks like the counters were more out of sync before the upgrade than after?
Do you know if your client is retrying counter operations ? (I saw some dropped messages in the S1 log).
S1 shows a lot of Commit Log replay going on. Reading your timeline below this sounds like the auto restart catching you out.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 16/11/2012, at 10:22 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Here is an example of the increase for some counter (counting events per hour)
>
> time (UTC) 0 1 2 3 4 5 6 7 8 9 10 11 12 13
> Good value 88 44 26 35 26 86 187 251 455 389 473 367 453 373
> C* counter 149 82 45 68 38 146 329 414 746 566 473 377 453 373
>
> I finished my Cassandra 1.1.6 upgrades at 9:30 UTC.
>
> I found wrong values since the day before at 20:00 UTC (counters from hours before are good)
>
> Here are the logs from the output:
> Server 1: http://pastebin.com/WyCm6Ef5 (This one is from the same server as the first bash history on my first mail)
> Server 2: http://pastebin.com/gBe2KL2b (This one is from the same server as the second bash history on my first mail)
>
> Alain
>
> 2012/11/15 aaron morton <aa...@thelastpickle.com>
> Can you provide an example of the increase ?
>
> Can you provide the log from startup ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
>> We had an issue with counters over-counting even using the nodetool drain command before upgrading...
>>
>> Here is my bash history
>>
>> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 70 cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>> 71 sudo apt-get install cassandra
>> 72 nodetool disablethrift
>> 73 nodetool drain
>> 74 service cassandra stop
>> 75 cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>> 76 vim /etc/cassandra/cassandra-env.sh
>> 77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 78 vim /etc/cassandra/cassandra.yaml
>> 79 service cassandra start
>>
>> So I think I followed these steps http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>>
>> I merged my conf files with an external tool so consider I merged my conf files on steps 76 and 78.
>>
>> I saw that the "sudo apt-get install cassandra" stop the server and restart it automatically. So it updated without draining and restart before I had the time to reconfigure the conf files. Is this "normal" ? Is there a way to avoid it ?
>>
>> So for the second node I decided to try to stop C*before the upgrade.
>>
>> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 126 cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>> 127 nodetool disablegossip
>> 128 nodetool disablethrift
>> 129 nodetool drain
>> 130 service cassandra stop
>> 131 sudo apt-get install cassandra
>>
>> //131 : This restarted cassandra
>>
>> 132 nodetool disablethrift
>> 133 nodetool disablegossip
>> 134 nodetool drain
>> 135 service cassandra stop
>> 136 cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>> 137 cim /etc/cassandra/cassandra-env.sh
>> 138 vim /etc/cassandra/cassandra-env.sh
>> 139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>> 140 vim /etc/cassandra/cassandra.yaml
>> 141 service cassandra start
>>
>> After both of these updates I saw my current counters increase without any reason.
>>
>> Did I do anything wrong ?
>>
>> Alain
>>
>
>
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Here is an example of the increase for some counter (counting events per
hour)
time (UTC) 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Good value 88 44 26 35 26 86 187 251 455 389 473 367 453 373
C* counter 149 82 45 68 38 146 329 414 746 566 473 377 453 373
I finished my Cassandra 1.1.6 upgrades at 9:30 UTC.
I found wrong values since the day before at 20:00 UTC (counters from hours
before are good)
Here are the logs from the output:
Server 1: http://pastebin.com/WyCm6Ef5 (This one is from the same server as
the first bash history on my first mail)
Server 2: http://pastebin.com/gBe2KL2b (This one is from the same server
as the second bash history on my first mail)
Alain
2012/11/15 aaron morton <aa...@thelastpickle.com>
> Can you provide an example of the increase ?
>
> Can you provide the log from startup ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> We had an issue with counters over-counting even using the nodetool drain
> command before upgrading...
>
> Here is my bash history
>
> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 70 cp /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 71 sudo apt-get install cassandra
> 72 nodetool disablethrift
> 73 nodetool drain
> 74 service cassandra stop
> 75 cat /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 76 vim /etc/cassandra/cassandra-env.sh
> 77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 78 vim /etc/cassandra/cassandra.yaml
> 79 service cassandra start
>
> So I think I followed these steps
> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>
> I merged my conf files with an external tool so consider I merged my conf
> files on steps 76 and 78.
>
> I saw that the "sudo apt-get install cassandra" stop the server and
> restart it automatically. So it updated without draining and restart before
> I had the time to reconfigure the conf files. Is this "normal" ? Is there a
> way to avoid it ?
>
> So for the second node I decided to try to stop C*before the upgrade.
>
> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 126 cp /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 127 nodetool disablegossip
> 128 nodetool disablethrift
> 129 nodetool drain
> 130 service cassandra stop
> 131 sudo apt-get install cassandra
>
> //131 : This restarted cassandra
>
> 132 nodetool disablethrift
> 133 nodetool disablegossip
> 134 nodetool drain
> 135 service cassandra stop
> 136 cat /etc/cassandra/cassandra-env.sh
> /etc/cassandra/cassandra-env.sh.bak
> 137 cim /etc/cassandra/cassandra-env.sh
> 138 vim /etc/cassandra/cassandra-env.sh
> 139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 140 vim /etc/cassandra/cassandra.yaml
> 141 service cassandra start
>
> After both of these updates I saw my current counters increase without any
> reason.
>
> Did I do anything wrong ?
>
> Alain
>
>
>
Re: Upgrade 1.1.2 -> 1.1.6
Posted by aaron morton <aa...@thelastpickle.com>.
Can you provide an example of the increase ?
Can you provide the log from startup ?
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> We had an issue with counters over-counting even using the nodetool drain command before upgrading...
>
> Here is my bash history
>
> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 70 cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
> 71 sudo apt-get install cassandra
> 72 nodetool disablethrift
> 73 nodetool drain
> 74 service cassandra stop
> 75 cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
> 76 vim /etc/cassandra/cassandra-env.sh
> 77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 78 vim /etc/cassandra/cassandra.yaml
> 79 service cassandra start
>
> So I think I followed these steps http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>
> I merged my conf files with an external tool so consider I merged my conf files on steps 76 and 78.
>
> I saw that the "sudo apt-get install cassandra" stop the server and restart it automatically. So it updated without draining and restart before I had the time to reconfigure the conf files. Is this "normal" ? Is there a way to avoid it ?
>
> So for the second node I decided to try to stop C*before the upgrade.
>
> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 126 cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
> 127 nodetool disablegossip
> 128 nodetool disablethrift
> 129 nodetool drain
> 130 service cassandra stop
> 131 sudo apt-get install cassandra
>
> //131 : This restarted cassandra
>
> 132 nodetool disablethrift
> 133 nodetool disablegossip
> 134 nodetool drain
> 135 service cassandra stop
> 136 cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
> 137 cim /etc/cassandra/cassandra-env.sh
> 138 vim /etc/cassandra/cassandra-env.sh
> 139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
> 140 vim /etc/cassandra/cassandra.yaml
> 141 service cassandra start
>
> After both of these updates I saw my current counters increase without any reason.
>
> Did I do anything wrong ?
>
> Alain
>
Re: Upgrade 1.1.2 -> 1.1.6
Posted by Rob Coli <rc...@palominodb.com>.
On Thu, Nov 15, 2012 at 6:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> We had an issue with counters over-counting even using the nodetool drain
> command before upgrading...
You're sure the over-count was caused by the upgrade?Counts can be
counted on (heh) to overcount. What is the scale of the over-count?
Also, do you realize that drain doesn't actually prevent over-replay,
though it should?
https://issues.apache.org/jira/browse/CASSANDRA-4446
Check for replayed operations in the system log..
"
grep -i replay /path/to/system.log
"
(I see further down thread that your logs do in fact contain replay.
Please comment to this effect, and how negative the experience is for
counts, on 4446?)
> I saw that the "sudo apt-get install cassandra" stop the server and restart
> it automatically. So it updated without draining and restart before I had
> the time to reconfigure the conf files. Is this "normal" ? Is there a way to
> avoid it ?
https://issues.apache.org/jira/browse/CASSANDRA-2356
Perhaps if you comment on here you can help convince the debian
packager that auto-starting a distributed database when you install or
upgrade its package has negative operational characteristics.
> After both of these updates I saw my current counters increase without any
> reason.
>
> Did I do anything wrong ?
Expecting counters to not over-count may qualify as "wrong." But your
process seems reasonable.
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb