You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stephen Baynes <st...@smoothwall.net> on 2015/10/02 16:45:33 UTC
Changing schema on multiple nodes while they are isolated
Is it safe to make schema changes ( e.g. create keyspace and tables ) on
multiple separate nodes of a cluster while they are out of communication
with other nodes in the cluster? For example create on node A while node B
is down, create on node B while A is down, then bring both up together.
We are looking to embed Cassandra invisibly in another product and we have
no control in what order users may start/stop the nodes up or add/remove
them from clusters. And Cassandra must come up and be working with at least
local access regardless. So this means always creating keyspaces and tables
so they are always present. But this means nodes joining clusters which
already have the same keyspace and table defined. Will it cause any issues?
I have done some testing and saw some some issues when I tried to nodetool
repair to bring things into sync. However at the time I was fighting with
what I later discovered was CASSANDRA-9689 keyspace does not show in
describe list, if create query times out.
<https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know
what was what. I will give it another try sometime, but would appreciate
knowing if this is going to run into trouble before we find it.
We are basically using Cassandra to share fairly transient information We
can cope with data loss during environment changes and occasional losses at
other times. But if the environment is stable then it should all just work,
whatever the environment is. We use a very high replication factor so all
nodes have a copy of all the data and will keep working even if they are
the only one up.
Thanks
--
Stephen Baynes
Re: Changing schema on multiple nodes while they are isolated
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Oct 2, 2015 at 9:08 AM, Stephen Baynes <
stephen.baynes@smoothwall.net> wrote:
> Will need to do some thinking about this. I wonder about shiping a backup
> of a good system keyspace and restore it on each node before it starts for
> the first time - but will that end up with each node having the same
> internal id?
>
(As a general statement, automated multi-threaded creation of schema in a
distributed database is a risky activity I do not advise... single thread
helps a lot...)
If you do a rolling restart, it will pick one of the two names.
Unfortunately it will then leave the SSTables from the loser name where
they are.
It's unclear whether restart with join_ring=false is enough to converge the
schemas...
But if so, you'd want to :
1) rolling restart nodes with join_ring=false so that their schema picks
the winner
2) stop the node and move the SSTables from the loser directory into the
winner directory (named appropriately, etc., watch out for name collision
etc.)
3) start the node with join_ring=false again
4) repair it
5) instruct the node to join the ring (or just restart it one more time)
=Rob
Re: Changing schema on multiple nodes while they are isolated
Posted by Clint Martin <cl...@coolfiretechnologies.com>.
No problem, glad to help. I'd love to see how it works out for you.
Clint
On Oct 5, 2015 8:12 AM, "Stephen Baynes" <st...@smoothwall.net>
wrote:
> Clint Martin's idea of each node creating its own keyspace, but then
> deciding which to use depending on the state of the cluster is really
> interesting. I am going to explore that in more detail.
>
> Thanks for the good idea.
>
> On 3 October 2015 at 00:03, Clint Martin <
> clintlmartin@coolfiretechnologies.com> wrote:
>
>> You could use a two key space method. At startup, wait some time for the
>> node to join the cluster.
>>
>> the first time the app starts, you can be in one of three states:
>>
>> The happiest state is that you succeed in joining a cluster. in this
>> case you will get replicated the cluster's keyspace and can start using it
>> as normal.
>>
>> the other two cases are exception cases: either you are the only node
>> ever to exist or you are a new node for a cluster that you cannot
>> communicate with. in either of these cases, you create a local private
>> copy of your schema in its own/unique keyspace.
>>
>> you application will use the "private" schema going forward until it
>> receives notification about another node joining the cluster. when this
>> occurs, your app attempts to create the "REAL" schema/keyspace (making
>> liberal use of "if not exists") and (if necessary) migrates the data from
>> it's "private" schema into the "real" schema followed by deleting the
>> "private" schema.
>>
>> There are edge cases and likely race conditions inherent to this method
>> that you would have to deal with, but it should do what you are describing.
>>
>> Clint
>> Hi Jacques-Henri
>>
>> You are right - serious trouble. I managed some more testing and it does
>> not repair or share any data. In the logs I see lots of:
>>
>> WARN [MessagingService-Incoming-/10.50.16.214] 2015-10-02 16:52:36,810
>> IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from
>> socket; closing
>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
>> cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163)
>> ~[apache-cassandra-2.2.1.jar:2.2.1]
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
>> ~[apache-cassandra-2.2.1.jar:2.2.1]
>>
>> and some:
>>
>> ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546
>> RepairMessageVerbHandler.java:164 - Got error, removing parent repair
>> session
>> ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548
>> CassandraDaemon.java:183 - Exception in thread
>> Thread[AntiEntropyStage:1,5,main]
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at
>> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167)
>> ~[apache-cassandra-2.2.1.jar:2.2.1]
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
>> ~[apache-cassandra-2.2.1.jar:2.2.1]
>>
>>
>> Will need to do some thinking about this. I wonder about shiping a backup
>> of a good system keyspace and restore it on each node before it starts for
>> the first time - but will that end up with each node having the same
>> internal id?
>>
>>
>>
>> On 2 October 2015 at 16:27, Jacques-Henri Berthemet <
>> jacques-henri.berthemet@genesys.com> wrote:
>>
>>> Hi Stephen,
>>>
>>>
>>>
>>> If you manage to create tables on each node while node A and B are
>>> separated, you’ll get into troubles when they will reconnect again. I had
>>> the case previously and Cassandra complained that tables with same names
>>> but different ids were present in the keyspace. I don’t know if there is a
>>> way to fix that with nodetool but I don’t think that it is a good practice.
>>>
>>>
>>>
>>> To solve this, we have a “schema creator” application node that is
>>> responsible to change the schema. If this node is down, schema updates are
>>> not possible. We can make any node ‘creator’, but only one can be enabled
>>> at any given time.
>>>
>>> *--*
>>>
>>> *Jacques-Henri Berthemet*
>>>
>>>
>>>
>>> *From:* Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
>>> *Sent:* vendredi 2 octobre 2015 16:46
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Changing schema on multiple nodes while they are isolated
>>>
>>>
>>>
>>> Is it safe to make schema changes ( e.g. create keyspace and tables ) on
>>> multiple separate nodes of a cluster while they are out of communication
>>> with other nodes in the cluster? For example create on node A while node B
>>> is down, create on node B while A is down, then bring both up together.
>>>
>>>
>>>
>>> We are looking to embed Cassandra invisibly in another product and we
>>> have no control in what order users may start/stop the nodes up or
>>> add/remove them from clusters. And Cassandra must come up and be working
>>> with at least local access regardless. So this means always creating
>>> keyspaces and tables so they are always present. But this means nodes
>>> joining clusters which already have the same keyspace and table defined.
>>> Will it cause any issues? I have done some testing and saw some some issues
>>> when I tried to nodetool repair to bring things into sync. However at the
>>> time I was fighting with what I later discovered was CASSANDRA-9689
>>> keyspace does not show in describe list, if create query times out.
>>> <https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know
>>> what was what. I will give it another try sometime, but would appreciate
>>> knowing if this is going to run into trouble before we find it.
>>>
>>>
>>>
>>> We are basically using Cassandra to share fairly transient information
>>> We can cope with data loss during environment changes and occasional losses
>>> at other times. But if the environment is stable then it should all just
>>> work, whatever the environment is. We use a very high replication factor so
>>> all nodes have a copy of all the data and will keep working even if they
>>> are the only one up.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> --
>>>
>>> *Stephen Baynes*
>>>
>>
>>
>> Thanks
>> --
>>
>> Stephen Baynes
>>
>
>
>
> --
>
> Stephen Baynes
>
Re: Changing schema on multiple nodes while they are isolated
Posted by Stephen Baynes <st...@smoothwall.net>.
Clint Martin's idea of each node creating its own keyspace, but then
deciding which to use depending on the state of the cluster is really
interesting. I am going to explore that in more detail.
Thanks for the good idea.
On 3 October 2015 at 00:03, Clint Martin <
clintlmartin@coolfiretechnologies.com> wrote:
> You could use a two key space method. At startup, wait some time for the
> node to join the cluster.
>
> the first time the app starts, you can be in one of three states:
>
> The happiest state is that you succeed in joining a cluster. in this case
> you will get replicated the cluster's keyspace and can start using it as
> normal.
>
> the other two cases are exception cases: either you are the only node ever
> to exist or you are a new node for a cluster that you cannot communicate
> with. in either of these cases, you create a local private copy of your
> schema in its own/unique keyspace.
>
> you application will use the "private" schema going forward until it
> receives notification about another node joining the cluster. when this
> occurs, your app attempts to create the "REAL" schema/keyspace (making
> liberal use of "if not exists") and (if necessary) migrates the data from
> it's "private" schema into the "real" schema followed by deleting the
> "private" schema.
>
> There are edge cases and likely race conditions inherent to this method
> that you would have to deal with, but it should do what you are describing.
>
> Clint
> Hi Jacques-Henri
>
> You are right - serious trouble. I managed some more testing and it does
> not repair or share any data. In the logs I see lots of:
>
> WARN [MessagingService-Incoming-/10.50.16.214] 2015-10-02 16:52:36,810
> IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from
> socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
> and some:
>
> ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546
> RepairMessageVerbHandler.java:164 - Got error, removing parent repair
> session
> ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548
> CassandraDaemon.java:183 - Exception in thread
> Thread[AntiEntropyStage:1,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
>
> Will need to do some thinking about this. I wonder about shiping a backup
> of a good system keyspace and restore it on each node before it starts for
> the first time - but will that end up with each node having the same
> internal id?
>
>
>
> On 2 October 2015 at 16:27, Jacques-Henri Berthemet <
> jacques-henri.berthemet@genesys.com> wrote:
>
>> Hi Stephen,
>>
>>
>>
>> If you manage to create tables on each node while node A and B are
>> separated, you’ll get into troubles when they will reconnect again. I had
>> the case previously and Cassandra complained that tables with same names
>> but different ids were present in the keyspace. I don’t know if there is a
>> way to fix that with nodetool but I don’t think that it is a good practice.
>>
>>
>>
>> To solve this, we have a “schema creator” application node that is
>> responsible to change the schema. If this node is down, schema updates are
>> not possible. We can make any node ‘creator’, but only one can be enabled
>> at any given time.
>>
>> *--*
>>
>> *Jacques-Henri Berthemet*
>>
>>
>>
>> *From:* Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
>> *Sent:* vendredi 2 octobre 2015 16:46
>> *To:* user@cassandra.apache.org
>> *Subject:* Changing schema on multiple nodes while they are isolated
>>
>>
>>
>> Is it safe to make schema changes ( e.g. create keyspace and tables ) on
>> multiple separate nodes of a cluster while they are out of communication
>> with other nodes in the cluster? For example create on node A while node B
>> is down, create on node B while A is down, then bring both up together.
>>
>>
>>
>> We are looking to embed Cassandra invisibly in another product and we
>> have no control in what order users may start/stop the nodes up or
>> add/remove them from clusters. And Cassandra must come up and be working
>> with at least local access regardless. So this means always creating
>> keyspaces and tables so they are always present. But this means nodes
>> joining clusters which already have the same keyspace and table defined.
>> Will it cause any issues? I have done some testing and saw some some issues
>> when I tried to nodetool repair to bring things into sync. However at the
>> time I was fighting with what I later discovered was CASSANDRA-9689
>> keyspace does not show in describe list, if create query times out.
>> <https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know
>> what was what. I will give it another try sometime, but would appreciate
>> knowing if this is going to run into trouble before we find it.
>>
>>
>>
>> We are basically using Cassandra to share fairly transient information We
>> can cope with data loss during environment changes and occasional losses at
>> other times. But if the environment is stable then it should all just work,
>> whatever the environment is. We use a very high replication factor so all
>> nodes have a copy of all the data and will keep working even if they are
>> the only one up.
>>
>>
>>
>> Thanks
>>
>>
>>
>> --
>>
>> *Stephen Baynes*
>>
>
>
> Thanks
> --
>
> Stephen Baynes
>
--
Stephen Baynes
Re: Changing schema on multiple nodes while they are isolated
Posted by Clint Martin <cl...@coolfiretechnologies.com>.
You could use a two key space method. At startup, wait some time for the
node to join the cluster.
the first time the app starts, you can be in one of three states:
The happiest state is that you succeed in joining a cluster. in this case
you will get replicated the cluster's keyspace and can start using it as
normal.
the other two cases are exception cases: either you are the only node ever
to exist or you are a new node for a cluster that you cannot communicate
with. in either of these cases, you create a local private copy of your
schema in its own/unique keyspace.
you application will use the "private" schema going forward until it
receives notification about another node joining the cluster. when this
occurs, your app attempts to create the "REAL" schema/keyspace (making
liberal use of "if not exists") and (if necessary) migrates the data from
it's "private" schema into the "real" schema followed by deleting the
"private" schema.
There are edge cases and likely race conditions inherent to this method
that you would have to deal with, but it should do what you are describing.
Clint
Hi Jacques-Henri
You are right - serious trouble. I managed some more testing and it does
not repair or share any data. In the logs I see lots of:
WARN [MessagingService-Incoming-/10.50.16.214] 2015-10-02 16:52:36,810
IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
~[apache-cassandra-2.2.1.jar:2.2.1]
and some:
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546
RepairMessageVerbHandler.java:164 - Got error, removing parent repair
session
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548 CassandraDaemon.java:183
- Exception in thread Thread[AntiEntropyStage:1,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-2.2.1.jar:2.2.1]
Will need to do some thinking about this. I wonder about shiping a backup
of a good system keyspace and restore it on each node before it starts for
the first time - but will that end up with each node having the same
internal id?
On 2 October 2015 at 16:27, Jacques-Henri Berthemet <
jacques-henri.berthemet@genesys.com> wrote:
> Hi Stephen,
>
>
>
> If you manage to create tables on each node while node A and B are
> separated, you’ll get into troubles when they will reconnect again. I had
> the case previously and Cassandra complained that tables with same names
> but different ids were present in the keyspace. I don’t know if there is a
> way to fix that with nodetool but I don’t think that it is a good practice.
>
>
>
> To solve this, we have a “schema creator” application node that is
> responsible to change the schema. If this node is down, schema updates are
> not possible. We can make any node ‘creator’, but only one can be enabled
> at any given time.
>
> *--*
>
> *Jacques-Henri Berthemet*
>
>
>
> *From:* Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
> *Sent:* vendredi 2 octobre 2015 16:46
> *To:* user@cassandra.apache.org
> *Subject:* Changing schema on multiple nodes while they are isolated
>
>
>
> Is it safe to make schema changes ( e.g. create keyspace and tables ) on
> multiple separate nodes of a cluster while they are out of communication
> with other nodes in the cluster? For example create on node A while node B
> is down, create on node B while A is down, then bring both up together.
>
>
>
> We are looking to embed Cassandra invisibly in another product and we have
> no control in what order users may start/stop the nodes up or add/remove
> them from clusters. And Cassandra must come up and be working with at least
> local access regardless. So this means always creating keyspaces and tables
> so they are always present. But this means nodes joining clusters which
> already have the same keyspace and table defined. Will it cause any issues?
> I have done some testing and saw some some issues when I tried to nodetool
> repair to bring things into sync. However at the time I was fighting with
> what I later discovered was CASSANDRA-9689 keyspace does not show in
> describe list, if create query times out.
> <https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know
> what was what. I will give it another try sometime, but would appreciate
> knowing if this is going to run into trouble before we find it.
>
>
>
> We are basically using Cassandra to share fairly transient information We
> can cope with data loss during environment changes and occasional losses at
> other times. But if the environment is stable then it should all just work,
> whatever the environment is. We use a very high replication factor so all
> nodes have a copy of all the data and will keep working even if they are
> the only one up.
>
>
>
> Thanks
>
>
>
> --
>
> *Stephen Baynes*
>
Thanks
--
Stephen Baynes
RE: Changing schema on multiple nodes while they are isolated
Posted by Jacques-Henri Berthemet <ja...@genesys.com>.
Then maybe Cassandra is not the right tool for that, or you need a different data structure. For example you could keep a single table where what used to be your table name is now a part of you partition key. That way any “offline” data will be merged when the nodes join again. If you have conflicts it will be resolved on the basis of the row timestamp.
--
Jacques-Henri Berthemet
From: Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
Sent: lundi 5 octobre 2015 11:00
To: user@cassandra.apache.org
Subject: Re: Changing schema on multiple nodes while they are isolated
> Why don’t you simply let the node join the cluster? It will pull new tables and the data automatically.
Because there is no guarantee the rest of the cluster is up, or even if there is anything more than a cluster of one at this time. This is a plug in and go environment where the user does not even know or care about the details of Cassandra. It is not a managed datacenter.
On 2 October 2015 at 17:16, Jacques-Henri Berthemet <ja...@genesys.com>> wrote:
Why don’t you simply let the node join the cluster? It will pull new tables and the data automatically.
--
Jacques-Henri Berthemet
From: Stephen Baynes [mailto:stephen.baynes@smoothwall.net<ma...@smoothwall.net>]
Sent: vendredi 2 octobre 2015 18:08
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Changing schema on multiple nodes while they are isolated
Hi Jacques-Henri
You are right - serious trouble. I managed some more testing and it does not repair or share any data. In the logs I see lots of:
WARN [MessagingService-Incoming-/10.50.16.214<http://10.50.16.214>] 2015-10-02 16:52:36,810 IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96) ~[apache-cassandra-2.2.1.jar:2.2.1]
and some:
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546 RepairMessageVerbHandler.java:164 - Got error, removing parent repair session
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548 CassandraDaemon.java:183 - Exception in thread Thread[AntiEntropyStage:1,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-2.2.1.jar:2.2.1]
Will need to do some thinking about this. I wonder about shiping a backup of a good system keyspace and restore it on each node before it starts for the first time - but will that end up with each node having the same internal id?
On 2 October 2015 at 16:27, Jacques-Henri Berthemet <ja...@genesys.com>> wrote:
Hi Stephen,
If you manage to create tables on each node while node A and B are separated, you’ll get into troubles when they will reconnect again. I had the case previously and Cassandra complained that tables with same names but different ids were present in the keyspace. I don’t know if there is a way to fix that with nodetool but I don’t think that it is a good practice.
To solve this, we have a “schema creator” application node that is responsible to change the schema. If this node is down, schema updates are not possible. We can make any node ‘creator’, but only one can be enabled at any given time.
--
Jacques-Henri Berthemet
From: Stephen Baynes [mailto:stephen.baynes@smoothwall.net<ma...@smoothwall.net>]
Sent: vendredi 2 octobre 2015 16:46
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Changing schema on multiple nodes while they are isolated
Is it safe to make schema changes ( e.g. create keyspace and tables ) on multiple separate nodes of a cluster while they are out of communication with other nodes in the cluster? For example create on node A while node B is down, create on node B while A is down, then bring both up together.
We are looking to embed Cassandra invisibly in another product and we have no control in what order users may start/stop the nodes up or add/remove them from clusters. And Cassandra must come up and be working with at least local access regardless. So this means always creating keyspaces and tables so they are always present. But this means nodes joining clusters which already have the same keyspace and table defined. Will it cause any issues? I have done some testing and saw some some issues when I tried to nodetool repair to bring things into sync. However at the time I was fighting with what I later discovered was CASSANDRA-9689 keyspace does not show in describe list, if create query times out.<https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know what was what. I will give it another try sometime, but would appreciate knowing if this is going to run into trouble before we find it.
We are basically using Cassandra to share fairly transient information We can cope with data loss during environment changes and occasional losses at other times. But if the environment is stable then it should all just work, whatever the environment is. We use a very high replication factor so all nodes have a copy of all the data and will keep working even if they are the only one up.
Thanks
--
Stephen Baynes
Thanks
--
Stephen Baynes
Thanks
--
Stephen Baynes
Senior Developer
Re: Changing schema on multiple nodes while they are isolated
Posted by Stephen Baynes <st...@smoothwall.net>.
> Why don’t you simply let the node join the cluster? It will pull new
tables and the data automatically.
Because there is no guarantee the rest of the cluster is up, or even if
there is anything more than a cluster of one at this time. This is a plug
in and go environment where the user does not even know or care about the
details of Cassandra. It is not a managed datacenter.
On 2 October 2015 at 17:16, Jacques-Henri Berthemet <
jacques-henri.berthemet@genesys.com> wrote:
> Why don’t you simply let the node join the cluster? It will pull new
> tables and the data automatically.
>
>
>
> *--*
>
> *Jacques-Henri Berthemet*
>
>
>
> *From:* Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
> *Sent:* vendredi 2 octobre 2015 18:08
> *To:* user@cassandra.apache.org
> *Subject:* Re: Changing schema on multiple nodes while they are isolated
>
>
>
> Hi Jacques-Henri
>
>
>
> You are right - serious trouble. I managed some more testing and it does
> not repair or share any data. In the logs I see lots of:
>
>
>
> WARN [MessagingService-Incoming-/10.50.16.214] 2015-10-02 16:52:36,810
> IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from
> socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
>
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
>
>
> and some:
>
>
>
> ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546
> RepairMessageVerbHandler.java:164 - Got error, removing parent repair
> session
>
> ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548
> CassandraDaemon.java:183 - Exception in thread
> Thread[AntiEntropyStage:1,5,main]
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
>
>
>
>
> Will need to do some thinking about this. I wonder about shiping a backup
> of a good system keyspace and restore it on each node before it starts for
> the first time - but will that end up with each node having the same
> internal id?
>
>
>
>
>
>
>
> On 2 October 2015 at 16:27, Jacques-Henri Berthemet <
> jacques-henri.berthemet@genesys.com> wrote:
>
> Hi Stephen,
>
>
>
> If you manage to create tables on each node while node A and B are
> separated, you’ll get into troubles when they will reconnect again. I had
> the case previously and Cassandra complained that tables with same names
> but different ids were present in the keyspace. I don’t know if there is a
> way to fix that with nodetool but I don’t think that it is a good practice.
>
>
>
> To solve this, we have a “schema creator” application node that is
> responsible to change the schema. If this node is down, schema updates are
> not possible. We can make any node ‘creator’, but only one can be enabled
> at any given time.
>
> *--*
>
> *Jacques-Henri Berthemet*
>
>
>
> *From:* Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
> *Sent:* vendredi 2 octobre 2015 16:46
> *To:* user@cassandra.apache.org
> *Subject:* Changing schema on multiple nodes while they are isolated
>
>
>
> Is it safe to make schema changes ( e.g. create keyspace and tables ) on
> multiple separate nodes of a cluster while they are out of communication
> with other nodes in the cluster? For example create on node A while node B
> is down, create on node B while A is down, then bring both up together.
>
>
>
> We are looking to embed Cassandra invisibly in another product and we have
> no control in what order users may start/stop the nodes up or add/remove
> them from clusters. And Cassandra must come up and be working with at least
> local access regardless. So this means always creating keyspaces and tables
> so they are always present. But this means nodes joining clusters which
> already have the same keyspace and table defined. Will it cause any issues?
> I have done some testing and saw some some issues when I tried to nodetool
> repair to bring things into sync. However at the time I was fighting with
> what I later discovered was CASSANDRA-9689 keyspace does not show in
> describe list, if create query times out.
> <https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know
> what was what. I will give it another try sometime, but would appreciate
> knowing if this is going to run into trouble before we find it.
>
>
>
> We are basically using Cassandra to share fairly transient information We
> can cope with data loss during environment changes and occasional losses at
> other times. But if the environment is stable then it should all just work,
> whatever the environment is. We use a very high replication factor so all
> nodes have a copy of all the data and will keep working even if they are
> the only one up.
>
>
>
> Thanks
>
>
>
> --
>
> *Stephen Baynes*
>
>
>
> Thanks
>
> --
>
> *Stephen Baynes*
>
Thanks
--
Stephen Baynes
Senior Developer
RE: Changing schema on multiple nodes while they are isolated
Posted by Jacques-Henri Berthemet <ja...@genesys.com>.
Why don’t you simply let the node join the cluster? It will pull new tables and the data automatically.
--
Jacques-Henri Berthemet
From: Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
Sent: vendredi 2 octobre 2015 18:08
To: user@cassandra.apache.org
Subject: Re: Changing schema on multiple nodes while they are isolated
Hi Jacques-Henri
You are right - serious trouble. I managed some more testing and it does not repair or share any data. In the logs I see lots of:
WARN [MessagingService-Incoming-/10.50.16.214<http://10.50.16.214>] 2015-10-02 16:52:36,810 IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96) ~[apache-cassandra-2.2.1.jar:2.2.1]
and some:
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546 RepairMessageVerbHandler.java:164 - Got error, removing parent repair session
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548 CassandraDaemon.java:183 - Exception in thread Thread[AntiEntropyStage:1,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167) ~[apache-cassandra-2.2.1.jar:2.2.1]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-2.2.1.jar:2.2.1]
Will need to do some thinking about this. I wonder about shiping a backup of a good system keyspace and restore it on each node before it starts for the first time - but will that end up with each node having the same internal id?
On 2 October 2015 at 16:27, Jacques-Henri Berthemet <ja...@genesys.com>> wrote:
Hi Stephen,
If you manage to create tables on each node while node A and B are separated, you’ll get into troubles when they will reconnect again. I had the case previously and Cassandra complained that tables with same names but different ids were present in the keyspace. I don’t know if there is a way to fix that with nodetool but I don’t think that it is a good practice.
To solve this, we have a “schema creator” application node that is responsible to change the schema. If this node is down, schema updates are not possible. We can make any node ‘creator’, but only one can be enabled at any given time.
--
Jacques-Henri Berthemet
From: Stephen Baynes [mailto:stephen.baynes@smoothwall.net<ma...@smoothwall.net>]
Sent: vendredi 2 octobre 2015 16:46
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Changing schema on multiple nodes while they are isolated
Is it safe to make schema changes ( e.g. create keyspace and tables ) on multiple separate nodes of a cluster while they are out of communication with other nodes in the cluster? For example create on node A while node B is down, create on node B while A is down, then bring both up together.
We are looking to embed Cassandra invisibly in another product and we have no control in what order users may start/stop the nodes up or add/remove them from clusters. And Cassandra must come up and be working with at least local access regardless. So this means always creating keyspaces and tables so they are always present. But this means nodes joining clusters which already have the same keyspace and table defined. Will it cause any issues? I have done some testing and saw some some issues when I tried to nodetool repair to bring things into sync. However at the time I was fighting with what I later discovered was CASSANDRA-9689 keyspace does not show in describe list, if create query times out.<https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know what was what. I will give it another try sometime, but would appreciate knowing if this is going to run into trouble before we find it.
We are basically using Cassandra to share fairly transient information We can cope with data loss during environment changes and occasional losses at other times. But if the environment is stable then it should all just work, whatever the environment is. We use a very high replication factor so all nodes have a copy of all the data and will keep working even if they are the only one up.
Thanks
--
Stephen Baynes
Thanks
--
Stephen Baynes
Re: Changing schema on multiple nodes while they are isolated
Posted by Stephen Baynes <st...@smoothwall.net>.
Hi Jacques-Henri
You are right - serious trouble. I managed some more testing and it does
not repair or share any data. In the logs I see lots of:
WARN [MessagingService-Incoming-/10.50.16.214] 2015-10-02 16:52:36,810
IncomingTcpConnection.java:100 - UnknownColumnFamilyException reading from
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=e6828dd0-691a-11e5-8a27-b1780df21c7c
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:163)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
~[apache-cassandra-2.2.1.jar:2.2.1]
and some:
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,546
RepairMessageVerbHandler.java:164 - Got error, removing parent repair
session
ERROR [AntiEntropyStage:1] 2015-10-02 16:48:16,548 CassandraDaemon.java:183
- Exception in thread Thread[AntiEntropyStage:1,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:167)
~[apache-cassandra-2.2.1.jar:2.2.1]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-2.2.1.jar:2.2.1]
Will need to do some thinking about this. I wonder about shiping a backup
of a good system keyspace and restore it on each node before it starts for
the first time - but will that end up with each node having the same
internal id?
On 2 October 2015 at 16:27, Jacques-Henri Berthemet <
jacques-henri.berthemet@genesys.com> wrote:
> Hi Stephen,
>
>
>
> If you manage to create tables on each node while node A and B are
> separated, you’ll get into troubles when they will reconnect again. I had
> the case previously and Cassandra complained that tables with same names
> but different ids were present in the keyspace. I don’t know if there is a
> way to fix that with nodetool but I don’t think that it is a good practice.
>
>
>
> To solve this, we have a “schema creator” application node that is
> responsible to change the schema. If this node is down, schema updates are
> not possible. We can make any node ‘creator’, but only one can be enabled
> at any given time.
>
> *--*
>
> *Jacques-Henri Berthemet*
>
>
>
> *From:* Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
> *Sent:* vendredi 2 octobre 2015 16:46
> *To:* user@cassandra.apache.org
> *Subject:* Changing schema on multiple nodes while they are isolated
>
>
>
> Is it safe to make schema changes ( e.g. create keyspace and tables ) on
> multiple separate nodes of a cluster while they are out of communication
> with other nodes in the cluster? For example create on node A while node B
> is down, create on node B while A is down, then bring both up together.
>
>
>
> We are looking to embed Cassandra invisibly in another product and we have
> no control in what order users may start/stop the nodes up or add/remove
> them from clusters. And Cassandra must come up and be working with at least
> local access regardless. So this means always creating keyspaces and tables
> so they are always present. But this means nodes joining clusters which
> already have the same keyspace and table defined. Will it cause any issues?
> I have done some testing and saw some some issues when I tried to nodetool
> repair to bring things into sync. However at the time I was fighting with
> what I later discovered was CASSANDRA-9689 keyspace does not show in
> describe list, if create query times out.
> <https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know
> what was what. I will give it another try sometime, but would appreciate
> knowing if this is going to run into trouble before we find it.
>
>
>
> We are basically using Cassandra to share fairly transient information We
> can cope with data loss during environment changes and occasional losses at
> other times. But if the environment is stable then it should all just work,
> whatever the environment is. We use a very high replication factor so all
> nodes have a copy of all the data and will keep working even if they are
> the only one up.
>
>
>
> Thanks
>
>
>
> --
>
> *Stephen Baynes*
>
Thanks
--
Stephen Baynes
RE: Changing schema on multiple nodes while they are isolated
Posted by Jacques-Henri Berthemet <ja...@genesys.com>.
Hi Stephen,
If you manage to create tables on each node while node A and B are separated, you’ll get into troubles when they will reconnect again. I had the case previously and Cassandra complained that tables with same names but different ids were present in the keyspace. I don’t know if there is a way to fix that with nodetool but I don’t think that it is a good practice.
To solve this, we have a “schema creator” application node that is responsible to change the schema. If this node is down, schema updates are not possible. We can make any node ‘creator’, but only one can be enabled at any given time.
--
Jacques-Henri Berthemet
From: Stephen Baynes [mailto:stephen.baynes@smoothwall.net]
Sent: vendredi 2 octobre 2015 16:46
To: user@cassandra.apache.org
Subject: Changing schema on multiple nodes while they are isolated
Is it safe to make schema changes ( e.g. create keyspace and tables ) on multiple separate nodes of a cluster while they are out of communication with other nodes in the cluster? For example create on node A while node B is down, create on node B while A is down, then bring both up together.
We are looking to embed Cassandra invisibly in another product and we have no control in what order users may start/stop the nodes up or add/remove them from clusters. And Cassandra must come up and be working with at least local access regardless. So this means always creating keyspaces and tables so they are always present. But this means nodes joining clusters which already have the same keyspace and table defined. Will it cause any issues? I have done some testing and saw some some issues when I tried to nodetool repair to bring things into sync. However at the time I was fighting with what I later discovered was CASSANDRA-9689 keyspace does not show in describe list, if create query times out.<https://issues.apache.org/jira/browse/CASSANDRA-9689> and did not know what was what. I will give it another try sometime, but would appreciate knowing if this is going to run into trouble before we find it.
We are basically using Cassandra to share fairly transient information We can cope with data loss during environment changes and occasional losses at other times. But if the environment is stable then it should all just work, whatever the environment is. We use a very high replication factor so all nodes have a copy of all the data and will keep working even if they are the only one up.
Thanks
--
Stephen Baynes