You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by mcasandra <mo...@gmail.com> on 2011/04/15 21:03:44 UTC
Two versions of schema
Is there a problem?
[default@StressKeyspace] update column family StressStandard with
keys_cached=100;
854ee0a0-6792-11e0-81f9-93d987913479
Waiting for schema agreement...
The schema has not settled in 10 seconds; further migrations are ill-advised
until it does.
Versions are 854ee0a0-6792-11e0-81f9-93d987913479:[10.18.62.202,
10.18.62.203, 10.18.62.200, 10.18.62.204, 10.18.62.199, 10.18.62.196,
10.18.62.197],22d165ff-6783-11e0-81f9-93d987913479:[10.18.62.198]
I remember reading somewhere before that when you have 2 versions of schemas
you are basically in trouble. Can someone explain what it means and it's
implications?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6277365.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Two versions of schema
Posted by mcasandra <mo...@gmail.com>.
I don't think I got correct answer to my original post. Can someone please
help?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6280070.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Two versions of schema
Posted by Peter Schuller <pe...@infidyne.com>.
> Schema changes should not be seen as something that can be done regularly. It should not be done programmatically. There should always be some operator looking at the cluster verifying that all nodes are reachable and ring is ok. And then issue schema changes one at a time using the cli.
+1. I think this is a great take-away w.r.t. schema changes.
--
/ Peter Schuller
Re: AW: AW: Two versions of schema
Posted by mcasandra <mo...@gmail.com>.
What would be the procedure in this case? Run drain on the node that is
disagreeing? But is it enough to run just drain or you suggest drain + rm
system files?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6287863.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
AW: AW: Two versions of schema
Posted by Roland Gude <ro...@yoochoose.com>.
Yeah it happens from time to time even if everything seems to be fine that schema changes don't work correctly. But it's always repairable with the described procedure. Therefore the operator being available is a must have I think.
Drain is a nodetool command. The node flushes data and stops accepting new writes. This just speeds up bringing the node back up again in this case. Probably a flush is equally acceptable.
-----Ursprüngliche Nachricht-----
Von: mcasandra [mailto:mohitanchlia@gmail.com]
Gesendet: Montag, 18. April 2011 18:27
An: cassandra-user@incubator.apache.org
Betreff: Re: AW: Two versions of schema
In my case all hosts were reachable and I ran nodetool ring before running
the schema update. I don't think it was because of node being down. I tihnk
for some reason it just took over 10 secs because I was reducing key_cache
from 1M to 1000. I think it might be taking long to trim the keys hence 10
sec default may not be the right way.
What is drain?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6284276.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: AW: Two versions of schema
Posted by mcasandra <mo...@gmail.com>.
In my case all hosts were reachable and I ran nodetool ring before running
the schema update. I don't think it was because of node being down. I tihnk
for some reason it just took over 10 secs because I was reducing key_cache
from 1M to 1000. I think it might be taking long to trim the keys hence 10
sec default may not be the right way.
What is drain?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6284276.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
AW: Two versions of schema
Posted by Roland Gude <ro...@yoochoose.com>.
Schema updates in cassandra tickle through the cluster over time very much like normal writes do.
But they keep some state indicating the "parent" schema and they will only be applied to some node if the parent schema is correct thus asserting the correct order of schema changes. This process is subject to failure for instance because of nodes being down or network errors. If such a failure happens there it's still possible that everything is relatively easy fixed. Take down the node that missed the updates (use drain) and bring it back only. This might work (and does not take that long). If it does not take it down again (again drain) delete the data in the system keyspace. Bring it back online.
There is however a situation where you are really screwed. It is when you issue updates concurrently to different nodes. The schema versions will disagree and there is now possibility to bring them back to the same version again. In this case you need to select one node, bring down all nodes that have a different schema, delete their system keyspace and bring them back.
Schema changes should not be seen as something that can be done regularly. It should not be done programmatically. There should always be some operator looking at the cluster verifying that all nodes are reachable and ring is ok. And then issue schema changes one at a time using the cli.
Greetings,
roland
-----Ursprüngliche Nachricht-----
Von: mcasandra [mailto:mohitanchlia@gmail.com]
Gesendet: Freitag, 15. April 2011 21:04
An: cassandra-user@incubator.apache.org
Betreff: Two versions of schema
Is there a problem?
[default@StressKeyspace] update column family StressStandard with
keys_cached=100;
854ee0a0-6792-11e0-81f9-93d987913479
Waiting for schema agreement...
The schema has not settled in 10 seconds; further migrations are ill-advised
until it does.
Versions are 854ee0a0-6792-11e0-81f9-93d987913479:[10.18.62.202,
10.18.62.203, 10.18.62.200, 10.18.62.204, 10.18.62.199, 10.18.62.196,
10.18.62.197],22d165ff-6783-11e0-81f9-93d987913479:[10.18.62.198]
I remember reading somewhere before that when you have 2 versions of schemas
you are basically in trouble. Can someone explain what it means and it's
implications?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6277365.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: cluster IP question and Jconsole?
Posted by Tyler Hobbs <ty...@datastax.com>.
See the first entry in http://wiki.apache.org/cassandra/JmxGotchas
On Wed, Apr 20, 2011 at 9:54 AM, tinhuty he <ti...@hotmail.com> wrote:
> Maki,
>
> Yes you are right, 8081 is mx4j port, the JMX_PORT is 8001 in the
> cassandra-env.sh.
>
> in the cassandra Linux server itself, I can run this successfully:
> nodetool -host xxxxx -p 8001 ring
> xxxxx is the actually IP address
>
> however when I run the same command in another windows machine(which has
> the cassandra windows version extracted), I am getting exception like below,
> one thing puzzled me is that the command trying to connect to ip xxxxx, but
> the exception claimed: "Connection refused to host: 127.0.0.1". Is there
> anything else that I need to config or...? I guess this is probably the
> reason that jconsole can't connect to port 8001 remotely either? Thanks for
> any advice!
>
> D:\apache-cassandra-0.7.4\bin>nodetool -host xxxxx -p 8001 ring
> Starting NodeTool
> Error connection to remote JMX agent!
> java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested
> exception is:
> java.net.ConnectException: Connection refused: connect
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)
> at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
> at sun.rmi.server.UnicastRef.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown
> Source)
> at javax.management.remote.rmi.RMIConnector.getConnection(Unknown
> Source)
> at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
> at javax.management.remote.JMXConnectorFactory.connect(Unknown
> Source)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
> Caused by: java.net.ConnectException: Connection refused: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(Unknown Source)
> at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
> at java.net.PlainSocketImpl.connect(Unknown Source)
> at java.net.SocksSocketImpl.connect(Unknown Source)
> at java.net.Socket.connect(Unknown Source)
> at java.net.Socket.connect(Unknown Source)
> at java.net.Socket.<init>(Unknown Source)
> at java.net.Socket.<init>(Unknown Source)
> at
> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source)
> at
> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source)
> ... 11 more
>
>
>
> -----Original Message----- From: Watanabe Maki
> Sent: Saturday, April 16, 2011 1:45 AM
>
> To: user@cassandra.apache.org
> Cc: <us...@cassandra.apache.org>
>
> Subject: Re: cluster IP question and Jconsole?
>
> 8081 is your mx4j port, isn't it? You need to connect jconsole to JMX_PORT
> specified in cassandra-env.sh.
>
> maki
>
> From iPhone
>
>
> On 2011/04/16, at 13:56, tinhuty he <ti...@hotmail.com> wrote:
>
> Maki, thanks for your reply. for the second question, I wasn't using the
>> loopback address, I was using the actually IP address for that server. I am
>> able to telnet to that IP on port 8081, but using jconsole failed.
>>
>> -----Original Message----- From: Maki Watanabe
>> Sent: Friday, April 15, 2011 9:43 PM
>> To: user@cassandra.apache.org
>> Cc: tinhuty he
>> Subject: Re: cluster IP question and Jconsole?
>>
>> 127.0.0.2 to 127.0.0.5 are valid IP addresses. Those are just alias
>> addresses for your loopback interface.
>> Verify:
>> % ifconfig -a
>>
>> 127.0.0.0/8 is for loopback, so you can't connect this address from
>> remote machines.
>> You may be able configure SSH port forwarding from your monitroing
>> host to cassandra node though I haven't try.
>>
>> maki
>>
>> 2011/4/16 tinhuty he <ti...@hotmail.com>:
>>
>>> I have followed the description here
>>>
>>> http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/lauching_5_node_cassandra_clusters
>>> to created 5 instances of cassandra in one CentOS 5.5 machine. using
>>> nodetool shows the 5 nodes are all running fine.
>>>
>>> Note the 5 nodes are using IP 127.0.0.1 to 127.0.0.5. I understand
>>> 127.0.0.1
>>> is pointing to local server, but how about 127.0.0.2 to 127.0.0.5? looks
>>> to
>>> me that they are not valid IP? how come all 5 nodes are working ok?
>>>
>>> Another question. I have installed MX4J in instance 127.0.0.1 on port
>>> 8081.
>>> I am able to connect to http://server:8081/ from the browser. However
>>> how do
>>> I connect using Jconsole that was installed in another windows
>>> machines?(since my CentOS5.5 doesn't have X installed, only SSH allowed).
>>>
>>> Thanks.
>>>
>>
>>
--
Tyler Hobbs
Software Engineer, DataStax <http://datastax.com/>
Maintainer of the pycassa <http://github.com/pycassa/pycassa> Cassandra
Python client library
Re: cluster IP question and Jconsole?
Posted by tinhuty he <ti...@hotmail.com>.
Maki,
Yes you are right, 8081 is mx4j port, the JMX_PORT is 8001 in the
cassandra-env.sh.
in the cassandra Linux server itself, I can run this successfully:
nodetool -host xxxxx -p 8001 ring
xxxxx is the actually IP address
however when I run the same command in another windows machine(which has the
cassandra windows version extracted), I am getting exception like below, one
thing puzzled me is that the command trying to connect to ip xxxxx, but the
exception claimed: "Connection refused to host: 127.0.0.1". Is there
anything else that I need to config or...? I guess this is probably the
reason that jconsole can't connect to port 8001 remotely either? Thanks for
any advice!
D:\apache-cassandra-0.7.4\bin>nodetool -host xxxxx -p 8001 ring
Starting NodeTool
Error connection to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested
exception is:
java.net.ConnectException: Connection refused: connect
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown
Source)
at javax.management.remote.rmi.RMIConnector.getConnection(Unknown
Source)
at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
at javax.management.remote.JMXConnectorFactory.connect(Unknown
Source)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source)
... 11 more
-----Original Message-----
From: Watanabe Maki
Sent: Saturday, April 16, 2011 1:45 AM
To: user@cassandra.apache.org
Cc: <us...@cassandra.apache.org>
Subject: Re: cluster IP question and Jconsole?
8081 is your mx4j port, isn't it? You need to connect jconsole to JMX_PORT
specified in cassandra-env.sh.
maki
>From iPhone
On 2011/04/16, at 13:56, tinhuty he <ti...@hotmail.com> wrote:
> Maki, thanks for your reply. for the second question, I wasn't using the
> loopback address, I was using the actually IP address for that server. I
> am able to telnet to that IP on port 8081, but using jconsole failed.
>
> -----Original Message----- From: Maki Watanabe
> Sent: Friday, April 15, 2011 9:43 PM
> To: user@cassandra.apache.org
> Cc: tinhuty he
> Subject: Re: cluster IP question and Jconsole?
>
> 127.0.0.2 to 127.0.0.5 are valid IP addresses. Those are just alias
> addresses for your loopback interface.
> Verify:
> % ifconfig -a
>
> 127.0.0.0/8 is for loopback, so you can't connect this address from
> remote machines.
> You may be able configure SSH port forwarding from your monitroing
> host to cassandra node though I haven't try.
>
> maki
>
> 2011/4/16 tinhuty he <ti...@hotmail.com>:
>> I have followed the description here
>> http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/lauching_5_node_cassandra_clusters
>> to created 5 instances of cassandra in one CentOS 5.5 machine. using
>> nodetool shows the 5 nodes are all running fine.
>>
>> Note the 5 nodes are using IP 127.0.0.1 to 127.0.0.5. I understand
>> 127.0.0.1
>> is pointing to local server, but how about 127.0.0.2 to 127.0.0.5? looks
>> to
>> me that they are not valid IP? how come all 5 nodes are working ok?
>>
>> Another question. I have installed MX4J in instance 127.0.0.1 on port
>> 8081.
>> I am able to connect to http://server:8081/ from the browser. However how
>> do
>> I connect using Jconsole that was installed in another windows
>> machines?(since my CentOS5.5 doesn't have X installed, only SSH allowed).
>>
>> Thanks.
>
Re: cluster IP question and Jconsole?
Posted by Watanabe Maki <wa...@gmail.com>.
8081 is your mx4j port, isn't it? You need to connect jconsole to JMX_PORT specified in cassandra-env.sh.
maki
From iPhone
On 2011/04/16, at 13:56, tinhuty he <ti...@hotmail.com> wrote:
> Maki, thanks for your reply. for the second question, I wasn't using the loopback address, I was using the actually IP address for that server. I am able to telnet to that IP on port 8081, but using jconsole failed.
>
> -----Original Message----- From: Maki Watanabe
> Sent: Friday, April 15, 2011 9:43 PM
> To: user@cassandra.apache.org
> Cc: tinhuty he
> Subject: Re: cluster IP question and Jconsole?
>
> 127.0.0.2 to 127.0.0.5 are valid IP addresses. Those are just alias
> addresses for your loopback interface.
> Verify:
> % ifconfig -a
>
> 127.0.0.0/8 is for loopback, so you can't connect this address from
> remote machines.
> You may be able configure SSH port forwarding from your monitroing
> host to cassandra node though I haven't try.
>
> maki
>
> 2011/4/16 tinhuty he <ti...@hotmail.com>:
>> I have followed the description here
>> http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/lauching_5_node_cassandra_clusters
>> to created 5 instances of cassandra in one CentOS 5.5 machine. using
>> nodetool shows the 5 nodes are all running fine.
>>
>> Note the 5 nodes are using IP 127.0.0.1 to 127.0.0.5. I understand 127.0.0.1
>> is pointing to local server, but how about 127.0.0.2 to 127.0.0.5? looks to
>> me that they are not valid IP? how come all 5 nodes are working ok?
>>
>> Another question. I have installed MX4J in instance 127.0.0.1 on port 8081.
>> I am able to connect to http://server:8081/ from the browser. However how do
>> I connect using Jconsole that was installed in another windows
>> machines?(since my CentOS5.5 doesn't have X installed, only SSH allowed).
>>
>> Thanks.
>
Re: cluster IP question and Jconsole?
Posted by tinhuty he <ti...@hotmail.com>.
Maki, thanks for your reply. for the second question, I wasn't using the
loopback address, I was using the actually IP address for that server. I am
able to telnet to that IP on port 8081, but using jconsole failed.
-----Original Message-----
From: Maki Watanabe
Sent: Friday, April 15, 2011 9:43 PM
To: user@cassandra.apache.org
Cc: tinhuty he
Subject: Re: cluster IP question and Jconsole?
127.0.0.2 to 127.0.0.5 are valid IP addresses. Those are just alias
addresses for your loopback interface.
Verify:
% ifconfig -a
127.0.0.0/8 is for loopback, so you can't connect this address from
remote machines.
You may be able configure SSH port forwarding from your monitroing
host to cassandra node though I haven't try.
maki
2011/4/16 tinhuty he <ti...@hotmail.com>:
> I have followed the description here
> http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/lauching_5_node_cassandra_clusters
> to created 5 instances of cassandra in one CentOS 5.5 machine. using
> nodetool shows the 5 nodes are all running fine.
>
> Note the 5 nodes are using IP 127.0.0.1 to 127.0.0.5. I understand
> 127.0.0.1
> is pointing to local server, but how about 127.0.0.2 to 127.0.0.5? looks
> to
> me that they are not valid IP? how come all 5 nodes are working ok?
>
> Another question. I have installed MX4J in instance 127.0.0.1 on port
> 8081.
> I am able to connect to http://server:8081/ from the browser. However how
> do
> I connect using Jconsole that was installed in another windows
> machines?(since my CentOS5.5 doesn't have X installed, only SSH allowed).
>
> Thanks.
Re: cluster IP question and Jconsole?
Posted by Maki Watanabe <wa...@gmail.com>.
127.0.0.2 to 127.0.0.5 are valid IP addresses. Those are just alias
addresses for your loopback interface.
Verify:
% ifconfig -a
127.0.0.0/8 is for loopback, so you can't connect this address from
remote machines.
You may be able configure SSH port forwarding from your monitroing
host to cassandra node though I haven't try.
maki
2011/4/16 tinhuty he <ti...@hotmail.com>:
> I have followed the description here
> http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/lauching_5_node_cassandra_clusters
> to created 5 instances of cassandra in one CentOS 5.5 machine. using
> nodetool shows the 5 nodes are all running fine.
>
> Note the 5 nodes are using IP 127.0.0.1 to 127.0.0.5. I understand 127.0.0.1
> is pointing to local server, but how about 127.0.0.2 to 127.0.0.5? looks to
> me that they are not valid IP? how come all 5 nodes are working ok?
>
> Another question. I have installed MX4J in instance 127.0.0.1 on port 8081.
> I am able to connect to http://server:8081/ from the browser. However how do
> I connect using Jconsole that was installed in another windows
> machines?(since my CentOS5.5 doesn't have X installed, only SSH allowed).
>
> Thanks.
cluster IP question and Jconsole?
Posted by tinhuty he <ti...@hotmail.com>.
I have followed the description here
http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/lauching_5_node_cassandra_clusters
to created 5 instances of cassandra in one CentOS 5.5 machine. using
nodetool shows the 5 nodes are all running fine.
Note the 5 nodes are using IP 127.0.0.1 to 127.0.0.5. I understand 127.0.0.1
is pointing to local server, but how about 127.0.0.2 to 127.0.0.5? looks to
me that they are not valid IP? how come all 5 nodes are working ok?
Another question. I have installed MX4J in instance 127.0.0.1 on port 8081.
I am able to connect to http://server:8081/ from the browser. However how do
I connect using Jconsole that was installed in another windows
machines?(since my CentOS5.5 doesn't have X installed, only SSH allowed).
Thanks.