You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David Aronchick <ar...@gmail.com> on 2015/07/01 02:16:05 UTC
Re: Issue when node goes away?
That is a GREAT lead! So it looks like I can't add a few nodes to the
cluster of the new version, have it settle down, and then upgrade the rest?
On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <ar...@gmail.com>
wrote:
> "Would it matter that I'm mixing cassandra versions? "
>
> From:
> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>
> "General upgrade limitations¶
>
> Do not run nodetool repair.
> Do not enable new features.
> Do not issue these types of queries during a rolling restart: DDL, TRUNCATE
> *During upgrades, the nodes on different versions show a schema
> disagreement*."
>
> I think this is a good lead.
>
> C*heers,
>
> Alain
>
> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>
>> I appreciate the thoughts! My issue is that it seems to work perfectly,
>> until the node goes away. Would it matter that I'm mixing cassandra
>> versions? (2.1.4 and 2.1.5)?
>>
>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <ar...@gmail.com>
>> wrote:
>>
>>> Hi David ?
>>>
>>> What does a "nodetool describecluster" output look like ?
>>>
>>> My guess is you might be having a schema version desynchronisation. If
>>> you see a node with different schema version you might want to try a
>>> "nodetool resetlocal*schema* - Reset node's local *schema* and
>>> resync"
>>>
>>> You asked for any thoughts, this is a thought, not sure if it will help,
>>> I hope so.
>>>
>>> C*heers,
>>>
>>> Alain
>>>
>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>
>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> Ping--- any thoughts here?
>>>>>
>>>>
>>>> I don't have any thoughts on your specific issue at this time, but FWIW
>>>> #cassandra on freenode is sometimes a better forum for interactive
>>>> debugging of operational edge cases.
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>>
>>
>
Re: Issue when node goes away?
Posted by David Aronchick <ar...@gmail.com>.
I understand - I was actually trying to use the containerized Cassandra, so
upgrading in place doesn't make sense.
I guess I'll wait until that's better supported.
On Wed, Jul 1, 2015 at 12:14 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Adding new nodes of a different version is asking for trouble, as others
> have said. I don't know if this particular version bump would expect any
> issues, but why risk it? If you want to upgrade, do it to the nodes in
> place. Don't bootstrap new nodes, don't run repair, don't remove nodes.
> My 2 cents.
>
> On Wed, Jul 1, 2015 at 11:59 AM David Aronchick <ar...@gmail.com>
> wrote:
>
>> I mean add an additional two nodes to my cluster and pointing them at the
>> other nodes in the cluster, to handle data migration.
>>
>> On Wed, Jul 1, 2015 at 11:40 AM, Jonathan Haddad <jo...@jonhaddad.com>
>> wrote:
>>
>>> When you say add 2 nodes, do you mean bootstrap, or upgrade in place?
>>>
>>> On Wed, Jul 1, 2015 at 11:37 AM David Aronchick <ar...@gmail.com>
>>> wrote:
>>>
>>>> This helps - so let me understand:
>>>>
>>>> Starting point:
>>>> - 4 nodes running 2.1.4
>>>> - System is healthy
>>>>
>>>> Decide to upgrade:
>>>> - Add 2 nodes running 2.1.5
>>>> - Run nodetool upgradestable
>>>> - Wait until system is healthy
>>>> - Stop 2 nodes running 2.1.4
>>>> - Run nodetool upgradestable
>>>> - Add 2 nodes running 2.1.5
>>>> - Run nodetool upgradestable
>>>> - Stop 2 nodes running 2.1.4
>>>> - Run nodetool upgradestable
>>>>
>>>> Finished?
>>>>
>>>> On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> Basically, when you add nodes, add them on the correct version to
>>>>> avoid schema / network issues in your streams. Also, try to update all the
>>>>> node using rolling restarts in a reduced time frame after stopping repairs,
>>>>> with all the nodes up, etc.
>>>>>
>>>>> You must have a healthy cluster before performing an upgrade.
>>>>>
>>>>> Remember running a nodetool upgradesstable after any upgrade (if not
>>>>> needed it will end fast, so I would run it always as a best practice, just
>>>>> in case).
>>>>>
>>>>> C*heers,
>>>>>
>>>>> Alain
>>>>>
>>>>> 2015-07-01 2:16 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>>>
>>>>>> That is a GREAT lead! So it looks like I can't add a few nodes to the
>>>>>> cluster of the new version, have it settle down, and then upgrade the rest?
>>>>>>
>>>>>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <arodrime@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> "Would it matter that I'm mixing cassandra versions? "
>>>>>>>
>>>>>>> From:
>>>>>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>>>>>>
>>>>>>> "General upgrade limitations¶
>>>>>>>
>>>>>>> Do not run nodetool repair.
>>>>>>> Do not enable new features.
>>>>>>> Do not issue these types of queries during a rolling restart: DDL,
>>>>>>> TRUNCATE
>>>>>>> *During upgrades, the nodes on different versions show a schema
>>>>>>> disagreement*."
>>>>>>>
>>>>>>> I think this is a good lead.
>>>>>>>
>>>>>>> C*heers,
>>>>>>>
>>>>>>> Alain
>>>>>>>
>>>>>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>>>>>
>>>>>>>> I appreciate the thoughts! My issue is that it seems to work
>>>>>>>> perfectly, until the node goes away. Would it matter that I'm mixing
>>>>>>>> cassandra versions? (2.1.4 and 2.1.5)?
>>>>>>>>
>>>>>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <
>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi David ?
>>>>>>>>>
>>>>>>>>> What does a "nodetool describecluster" output look like ?
>>>>>>>>>
>>>>>>>>> My guess is you might be having a schema version
>>>>>>>>> desynchronisation. If you see a node with different schema version you
>>>>>>>>> might want to try a "nodetool resetlocal*schema* - Reset
>>>>>>>>> node's local *schema* and resync"
>>>>>>>>>
>>>>>>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>>>>>>> help, I hope so.
>>>>>>>>>
>>>>>>>>> C*heers,
>>>>>>>>>
>>>>>>>>> Alain
>>>>>>>>>
>>>>>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>>>>>>>
>>>>>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <
>>>>>>>>>> aronchick@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ping--- any thoughts here?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don't have any thoughts on your specific issue at this time,
>>>>>>>>>> but FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>>>>>>> debugging of operational edge cases.
>>>>>>>>>>
>>>>>>>>>> =Rob
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
Re: Issue when node goes away?
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Jul 1, 2015 at 12:14 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Adding new nodes of a different version is asking for trouble, as others
> have said. I don't know if this particular version bump would expect any
> issues, but why risk it? If you want to upgrade, do it to the nodes in
> place. Don't bootstrap new nodes, don't run repair, don't remove nodes.
> My 2 cents.
>
If someone else adds in a penny, we'll have a nickle. :D
=Rob
Re: Issue when node goes away?
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Adding new nodes of a different version is asking for trouble, as others
have said. I don't know if this particular version bump would expect any
issues, but why risk it? If you want to upgrade, do it to the nodes in
place. Don't bootstrap new nodes, don't run repair, don't remove nodes.
My 2 cents.
On Wed, Jul 1, 2015 at 11:59 AM David Aronchick <ar...@gmail.com> wrote:
> I mean add an additional two nodes to my cluster and pointing them at the
> other nodes in the cluster, to handle data migration.
>
> On Wed, Jul 1, 2015 at 11:40 AM, Jonathan Haddad <jo...@jonhaddad.com>
> wrote:
>
>> When you say add 2 nodes, do you mean bootstrap, or upgrade in place?
>>
>> On Wed, Jul 1, 2015 at 11:37 AM David Aronchick <ar...@gmail.com>
>> wrote:
>>
>>> This helps - so let me understand:
>>>
>>> Starting point:
>>> - 4 nodes running 2.1.4
>>> - System is healthy
>>>
>>> Decide to upgrade:
>>> - Add 2 nodes running 2.1.5
>>> - Run nodetool upgradestable
>>> - Wait until system is healthy
>>> - Stop 2 nodes running 2.1.4
>>> - Run nodetool upgradestable
>>> - Add 2 nodes running 2.1.5
>>> - Run nodetool upgradestable
>>> - Stop 2 nodes running 2.1.4
>>> - Run nodetool upgradestable
>>>
>>> Finished?
>>>
>>> On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>> wrote:
>>>
>>>> Basically, when you add nodes, add them on the correct version to avoid
>>>> schema / network issues in your streams. Also, try to update all the node
>>>> using rolling restarts in a reduced time frame after stopping repairs, with
>>>> all the nodes up, etc.
>>>>
>>>> You must have a healthy cluster before performing an upgrade.
>>>>
>>>> Remember running a nodetool upgradesstable after any upgrade (if not
>>>> needed it will end fast, so I would run it always as a best practice, just
>>>> in case).
>>>>
>>>> C*heers,
>>>>
>>>> Alain
>>>>
>>>> 2015-07-01 2:16 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>>
>>>>> That is a GREAT lead! So it looks like I can't add a few nodes to the
>>>>> cluster of the new version, have it settle down, and then upgrade the rest?
>>>>>
>>>>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> "Would it matter that I'm mixing cassandra versions? "
>>>>>>
>>>>>> From:
>>>>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>>>>>
>>>>>> "General upgrade limitations¶
>>>>>>
>>>>>> Do not run nodetool repair.
>>>>>> Do not enable new features.
>>>>>> Do not issue these types of queries during a rolling restart: DDL,
>>>>>> TRUNCATE
>>>>>> *During upgrades, the nodes on different versions show a schema
>>>>>> disagreement*."
>>>>>>
>>>>>> I think this is a good lead.
>>>>>>
>>>>>> C*heers,
>>>>>>
>>>>>> Alain
>>>>>>
>>>>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>>>>
>>>>>>> I appreciate the thoughts! My issue is that it seems to work
>>>>>>> perfectly, until the node goes away. Would it matter that I'm mixing
>>>>>>> cassandra versions? (2.1.4 and 2.1.5)?
>>>>>>>
>>>>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <arodrime@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi David ?
>>>>>>>>
>>>>>>>> What does a "nodetool describecluster" output look like ?
>>>>>>>>
>>>>>>>> My guess is you might be having a schema version desynchronisation.
>>>>>>>> If you see a node with different schema version you might want to try a
>>>>>>>> "nodetool resetlocal*schema* - Reset node's local *schema*
>>>>>>>> and resync"
>>>>>>>>
>>>>>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>>>>>> help, I hope so.
>>>>>>>>
>>>>>>>> C*heers,
>>>>>>>>
>>>>>>>> Alain
>>>>>>>>
>>>>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>>>>>>
>>>>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <
>>>>>>>>> aronchick@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Ping--- any thoughts here?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I don't have any thoughts on your specific issue at this time, but
>>>>>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>>>>>> debugging of operational edge cases.
>>>>>>>>>
>>>>>>>>> =Rob
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Issue when node goes away?
Posted by David Aronchick <ar...@gmail.com>.
I mean add an additional two nodes to my cluster and pointing them at the
other nodes in the cluster, to handle data migration.
On Wed, Jul 1, 2015 at 11:40 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> When you say add 2 nodes, do you mean bootstrap, or upgrade in place?
>
> On Wed, Jul 1, 2015 at 11:37 AM David Aronchick <ar...@gmail.com>
> wrote:
>
>> This helps - so let me understand:
>>
>> Starting point:
>> - 4 nodes running 2.1.4
>> - System is healthy
>>
>> Decide to upgrade:
>> - Add 2 nodes running 2.1.5
>> - Run nodetool upgradestable
>> - Wait until system is healthy
>> - Stop 2 nodes running 2.1.4
>> - Run nodetool upgradestable
>> - Add 2 nodes running 2.1.5
>> - Run nodetool upgradestable
>> - Stop 2 nodes running 2.1.4
>> - Run nodetool upgradestable
>>
>> Finished?
>>
>> On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <ar...@gmail.com>
>> wrote:
>>
>>> Basically, when you add nodes, add them on the correct version to avoid
>>> schema / network issues in your streams. Also, try to update all the node
>>> using rolling restarts in a reduced time frame after stopping repairs, with
>>> all the nodes up, etc.
>>>
>>> You must have a healthy cluster before performing an upgrade.
>>>
>>> Remember running a nodetool upgradesstable after any upgrade (if not
>>> needed it will end fast, so I would run it always as a best practice, just
>>> in case).
>>>
>>> C*heers,
>>>
>>> Alain
>>>
>>> 2015-07-01 2:16 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>
>>>> That is a GREAT lead! So it looks like I can't add a few nodes to the
>>>> cluster of the new version, have it settle down, and then upgrade the rest?
>>>>
>>>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> "Would it matter that I'm mixing cassandra versions? "
>>>>>
>>>>> From:
>>>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>>>>
>>>>> "General upgrade limitations¶
>>>>>
>>>>> Do not run nodetool repair.
>>>>> Do not enable new features.
>>>>> Do not issue these types of queries during a rolling restart: DDL,
>>>>> TRUNCATE
>>>>> *During upgrades, the nodes on different versions show a schema
>>>>> disagreement*."
>>>>>
>>>>> I think this is a good lead.
>>>>>
>>>>> C*heers,
>>>>>
>>>>> Alain
>>>>>
>>>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>>>
>>>>>> I appreciate the thoughts! My issue is that it seems to work
>>>>>> perfectly, until the node goes away. Would it matter that I'm mixing
>>>>>> cassandra versions? (2.1.4 and 2.1.5)?
>>>>>>
>>>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi David ?
>>>>>>>
>>>>>>> What does a "nodetool describecluster" output look like ?
>>>>>>>
>>>>>>> My guess is you might be having a schema version desynchronisation.
>>>>>>> If you see a node with different schema version you might want to try a
>>>>>>> "nodetool resetlocal*schema* - Reset node's local *schema*
>>>>>>> and resync"
>>>>>>>
>>>>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>>>>> help, I hope so.
>>>>>>>
>>>>>>> C*heers,
>>>>>>>
>>>>>>> Alain
>>>>>>>
>>>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>>>>>
>>>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <
>>>>>>>> aronchick@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Ping--- any thoughts here?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't have any thoughts on your specific issue at this time, but
>>>>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>>>>> debugging of operational edge cases.
>>>>>>>>
>>>>>>>> =Rob
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Issue when node goes away?
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
When you say add 2 nodes, do you mean bootstrap, or upgrade in place?
On Wed, Jul 1, 2015 at 11:37 AM David Aronchick <ar...@gmail.com> wrote:
> This helps - so let me understand:
>
> Starting point:
> - 4 nodes running 2.1.4
> - System is healthy
>
> Decide to upgrade:
> - Add 2 nodes running 2.1.5
> - Run nodetool upgradestable
> - Wait until system is healthy
> - Stop 2 nodes running 2.1.4
> - Run nodetool upgradestable
> - Add 2 nodes running 2.1.5
> - Run nodetool upgradestable
> - Stop 2 nodes running 2.1.4
> - Run nodetool upgradestable
>
> Finished?
>
> On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
>
>> Basically, when you add nodes, add them on the correct version to avoid
>> schema / network issues in your streams. Also, try to update all the node
>> using rolling restarts in a reduced time frame after stopping repairs, with
>> all the nodes up, etc.
>>
>> You must have a healthy cluster before performing an upgrade.
>>
>> Remember running a nodetool upgradesstable after any upgrade (if not
>> needed it will end fast, so I would run it always as a best practice, just
>> in case).
>>
>> C*heers,
>>
>> Alain
>>
>> 2015-07-01 2:16 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>
>>> That is a GREAT lead! So it looks like I can't add a few nodes to the
>>> cluster of the new version, have it settle down, and then upgrade the rest?
>>>
>>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>> wrote:
>>>
>>>> "Would it matter that I'm mixing cassandra versions? "
>>>>
>>>> From:
>>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>>>
>>>> "General upgrade limitations¶
>>>>
>>>> Do not run nodetool repair.
>>>> Do not enable new features.
>>>> Do not issue these types of queries during a rolling restart: DDL,
>>>> TRUNCATE
>>>> *During upgrades, the nodes on different versions show a schema
>>>> disagreement*."
>>>>
>>>> I think this is a good lead.
>>>>
>>>> C*heers,
>>>>
>>>> Alain
>>>>
>>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>>
>>>>> I appreciate the thoughts! My issue is that it seems to work
>>>>> perfectly, until the node goes away. Would it matter that I'm mixing
>>>>> cassandra versions? (2.1.4 and 2.1.5)?
>>>>>
>>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi David ?
>>>>>>
>>>>>> What does a "nodetool describecluster" output look like ?
>>>>>>
>>>>>> My guess is you might be having a schema version desynchronisation.
>>>>>> If you see a node with different schema version you might want to try a
>>>>>> "nodetool resetlocal*schema* - Reset node's local *schema* and
>>>>>> resync"
>>>>>>
>>>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>>>> help, I hope so.
>>>>>>
>>>>>> C*heers,
>>>>>>
>>>>>> Alain
>>>>>>
>>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>>>>
>>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <
>>>>>>> aronchick@gmail.com> wrote:
>>>>>>>
>>>>>>>> Ping--- any thoughts here?
>>>>>>>>
>>>>>>>
>>>>>>> I don't have any thoughts on your specific issue at this time, but
>>>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>>>> debugging of operational edge cases.
>>>>>>>
>>>>>>> =Rob
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Issue when node goes away?
Posted by David Aronchick <ar...@gmail.com>.
This helps - so let me understand:
Starting point:
- 4 nodes running 2.1.4
- System is healthy
Decide to upgrade:
- Add 2 nodes running 2.1.5
- Run nodetool upgradestable
- Wait until system is healthy
- Stop 2 nodes running 2.1.4
- Run nodetool upgradestable
- Add 2 nodes running 2.1.5
- Run nodetool upgradestable
- Stop 2 nodes running 2.1.4
- Run nodetool upgradestable
Finished?
On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Basically, when you add nodes, add them on the correct version to avoid
> schema / network issues in your streams. Also, try to update all the node
> using rolling restarts in a reduced time frame after stopping repairs, with
> all the nodes up, etc.
>
> You must have a healthy cluster before performing an upgrade.
>
> Remember running a nodetool upgradesstable after any upgrade (if not
> needed it will end fast, so I would run it always as a best practice, just
> in case).
>
> C*heers,
>
> Alain
>
> 2015-07-01 2:16 GMT+02:00 David Aronchick <ar...@gmail.com>:
>
>> That is a GREAT lead! So it looks like I can't add a few nodes to the
>> cluster of the new version, have it settle down, and then upgrade the rest?
>>
>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <ar...@gmail.com>
>> wrote:
>>
>>> "Would it matter that I'm mixing cassandra versions? "
>>>
>>> From:
>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>>
>>> "General upgrade limitations¶
>>>
>>> Do not run nodetool repair.
>>> Do not enable new features.
>>> Do not issue these types of queries during a rolling restart: DDL,
>>> TRUNCATE
>>> *During upgrades, the nodes on different versions show a schema
>>> disagreement*."
>>>
>>> I think this is a good lead.
>>>
>>> C*heers,
>>>
>>> Alain
>>>
>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>>
>>>> I appreciate the thoughts! My issue is that it seems to work perfectly,
>>>> until the node goes away. Would it matter that I'm mixing cassandra
>>>> versions? (2.1.4 and 2.1.5)?
>>>>
>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi David ?
>>>>>
>>>>> What does a "nodetool describecluster" output look like ?
>>>>>
>>>>> My guess is you might be having a schema version desynchronisation. If
>>>>> you see a node with different schema version you might want to try a
>>>>> "nodetool resetlocal*schema* - Reset node's local *schema* and
>>>>> resync"
>>>>>
>>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>>> help, I hope so.
>>>>>
>>>>> C*heers,
>>>>>
>>>>> Alain
>>>>>
>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>>>
>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <aronchick@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Ping--- any thoughts here?
>>>>>>>
>>>>>>
>>>>>> I don't have any thoughts on your specific issue at this time, but
>>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>>> debugging of operational edge cases.
>>>>>>
>>>>>> =Rob
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Issue when node goes away?
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Basically, when you add nodes, add them on the correct version to avoid
schema / network issues in your streams. Also, try to update all the node
using rolling restarts in a reduced time frame after stopping repairs, with
all the nodes up, etc.
You must have a healthy cluster before performing an upgrade.
Remember running a nodetool upgradesstable after any upgrade (if not needed
it will end fast, so I would run it always as a best practice, just in
case).
C*heers,
Alain
2015-07-01 2:16 GMT+02:00 David Aronchick <ar...@gmail.com>:
> That is a GREAT lead! So it looks like I can't add a few nodes to the
> cluster of the new version, have it settle down, and then upgrade the rest?
>
> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
>
>> "Would it matter that I'm mixing cassandra versions? "
>>
>> From:
>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>
>> "General upgrade limitations¶
>>
>> Do not run nodetool repair.
>> Do not enable new features.
>> Do not issue these types of queries during a rolling restart: DDL,
>> TRUNCATE
>> *During upgrades, the nodes on different versions show a schema
>> disagreement*."
>>
>> I think this is a good lead.
>>
>> C*heers,
>>
>> Alain
>>
>> 2015-06-30 20:22 GMT+02:00 David Aronchick <ar...@gmail.com>:
>>
>>> I appreciate the thoughts! My issue is that it seems to work perfectly,
>>> until the node goes away. Would it matter that I'm mixing cassandra
>>> versions? (2.1.4 and 2.1.5)?
>>>
>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>> wrote:
>>>
>>>> Hi David ?
>>>>
>>>> What does a "nodetool describecluster" output look like ?
>>>>
>>>> My guess is you might be having a schema version desynchronisation. If
>>>> you see a node with different schema version you might want to try a
>>>> "nodetool resetlocal*schema* - Reset node's local *schema* and
>>>> resync"
>>>>
>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>> help, I hope so.
>>>>
>>>> C*heers,
>>>>
>>>> Alain
>>>>
>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rc...@eventbrite.com>:
>>>>
>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <ar...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ping--- any thoughts here?
>>>>>>
>>>>>
>>>>> I don't have any thoughts on your specific issue at this time, but
>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>> debugging of operational edge cases.
>>>>>
>>>>> =Rob
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>