You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David Boxenhorn <da...@lookin2.com> on 2010/06/02 12:18:10 UTC
Heterogeneous Cassandra Cluster
Is it possible to make a heterogeneous Cassandra cluster, with both Linux
and Windows nodes? I tried doing it and got
Error in ThreadPoolExecutor
java.lang.NullPointerException
Not sure if this is due to the Linux/Windows mix or something else.
Details below:
[root@IQDEV01 cassandra]# bin/cassandra -f
INFO 20:32:26,431 Auto DiskAccessMode determined to be mmap
INFO 20:32:27,085 Sampling index for
/var/lib/cassandra/data/system/LocationInfo-1-Data.db
INFO 20:32:27,095 Sampling index for
/var/lib/cassandra/data/system/LocationInfo-2-Data.db
INFO 20:32:27,104 Replaying
/var/lib/cassandra/commitlog/CommitLog-1275412410865.log
INFO 20:32:27,129 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1275413547129.log
INFO 20:32:27,138 LocationInfo has reached its threshold; switching in a
fresh Memtable at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1275413547129.log',
position=173)
INFO 20:32:27,138 Enqueuing flush of Memtable(LocationInfo)@1491010616
INFO 20:32:27,139 Writing Memtable(LocationInfo)@1491010616
INFO 20:32:27,187 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-3-Data.db
INFO 20:32:27,207 Log replay complete
INFO 20:32:27,239 Saved Token found: 25870423804996813139937576731363583348
INFO 20:32:27,239 Saved ClusterName found: Lookin2
INFO 20:32:27,247 Starting up server gossip
INFO 20:32:27,266 Joining: getting load information
INFO 20:32:27,267 Sleeping 90000 ms to wait for load information...
INFO 20:32:27,327 Node /192.168.80.12 is now part of the cluster
INFO 20:32:27,332 Node /192.168.80.234 is now part of the cluster
INFO 20:32:27,864 InetAddress /192.168.80.12 is now UP
INFO 20:32:27,872 InetAddress /192.168.80.234 is now UP
INFO 20:33:57,269 Joining: getting bootstrap token
INFO 20:33:57,278 New token will be 25870423804996813139937576731363583348
to assume load from /192.168.80.12
INFO 20:33:57,279 Joining: sleeping 30000 for pending range setup
INFO 20:34:27,280 Bootstrapping
INFO 21:32:27,867 Compacting []
INFO 21:38:27,118 LocationInfo has reached its threshold; switching in a
fresh Memtable at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1275413547129.log',
position=824)
INFO 21:38:27,118 Enqueuing flush of Memtable(LocationInfo)@993374707
INFO 21:38:27,118 Writing Memtable(LocationInfo)@993374707
INFO 21:38:27,158 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-4-Data.db
INFO 21:38:27,160 Compacting
[org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-1-Data.db'),org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-2-Data.db'),org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-3-Data.db'),org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-4-Data.db')]
INFO 21:38:27,217 Compacted to
/var/lib/cassandra/data/system/LocationInfo-5-Data.db. 1294/358 bytes for 1
keys. Time: 56ms.
[root@IQDEV01 cassandra]# bin/cassandra -f
INFO 21:40:07,519 Auto DiskAccessMode determined to be mmap
INFO 21:40:07,972 Deleted
/var/lib/cassandra/data/system/LocationInfo-1-Data.db
INFO 21:40:07,973 Deleted
/var/lib/cassandra/data/system/LocationInfo-2-Data.db
INFO 21:40:07,974 Deleted
/var/lib/cassandra/data/system/LocationInfo-3-Data.db
INFO 21:40:07,982 Sampling index for
/var/lib/cassandra/data/system/LocationInfo-5-Data.db
INFO 21:40:07,991 Deleted
/var/lib/cassandra/data/system/LocationInfo-4-Data.db
INFO 21:40:08,000 Replaying
/var/lib/cassandra/commitlog/CommitLog-1275413547129.log
INFO 21:40:08,001 Log replay complete
INFO 21:40:08,038 Saved Token found: 25870423804996813139937576731363583348
INFO 21:40:08,040 Saved ClusterName found: Lookin2
INFO 21:40:08,042 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1275417608042.log
INFO 21:40:08,059 Starting up server gossip
INFO 21:40:08,071 Joining: getting load information
INFO 21:40:08,071 Sleeping 90000 ms to wait for load information...
INFO 21:40:10,372 Node /192.168.80.12 is now part of the cluster
INFO 21:40:10,374 Node /192.168.80.234 is now part of the cluster
INFO 21:40:11,091 InetAddress /192.168.80.234 is now UP
INFO 21:40:12,078 InetAddress /192.168.80.12 is now UP
INFO 21:41:38,072 Joining: getting bootstrap token
INFO 21:41:38,088 New token will be 25870423804996813139937576731363583348
to assume load from /192.168.80.12
INFO 21:41:38,089 Joining: sleeping 30000 for pending range setup
INFO 21:42:08,091 Bootstrapping
ERROR 21:49:03,526 Error in ThreadPoolExecutor
java.lang.NullPointerException
at
org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
at
org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR 21:49:03,527 Fatal exception in thread
Thread[MESSAGE-DESERIALIZER-POOL:1,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
at
org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO 22:40:08,015 LocationInfo has reached its threshold; switching in a
fresh Memtable at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1275417608042.log',
position=824)
INFO 22:40:08,016 Enqueuing flush of Memtable(LocationInfo)@925668695
INFO 22:40:08,017 Writing Memtable(LocationInfo)@925668695
INFO 22:40:08,058 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-6-Data.db
INFO 22:40:11,100 Compacting []
Re: Heterogeneous Cassandra Cluster
Posted by David Boxenhorn <da...@lookin2.com>.
Our replication factor was 1, so that wasn't the problem. (We tried other
replication factors too, just in case, but it didn't help.)
On Wed, Jun 2, 2010 at 7:51 PM, Nahor
<na...@gmail.com>
> wrote:
> On 2010-06-02 3:18, David Boxenhorn wrote:
>
>> Is it possible to make a heterogeneous Cassandra cluster, with both Linux
>> and Windows nodes? I tried doing it and got
>>
>> Error in ThreadPoolExecutor
>> java.lang.NullPointerException
>>
>> Not sure if this is due to the Linux/Windows mix or something else.
>>
>>
>> Details below:
>>
> [...]
>
>
>> INFO 21:42:08,091 Bootstrapping
>>
>> ERROR 21:49:03,526 Error in ThreadPoolExecutor
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
>>
>> at
>> org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
>>
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> at java.lang.Thread.run(Thread.java:619)
>>
>> ERROR 21:49:03,527 Fatal exception in thread
>> Thread[MESSAGE-DESERIALIZER-POOL:1,5,main]
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
>>
>> at
>> org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
>>
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>>
> Looks like https://issues.apache.org/jira/browse/CASSANDRA-1136
>
> Make sure you have enough nodes in your cluster to satisfy your replication
> factor before you add any data. This is what seems to be source of the
> problem in my case.
>
> That said, I was also using an heterogeneous system (Linux + Windows) but I
> think I tested it with only Linux nodes too.
>
>
>
Re: Heterogeneous Cassandra Cluster
Posted by Nahor <na...@gmail.com>.
On 2010-06-02 3:18, David Boxenhorn wrote:
> Is it possible to make a heterogeneous Cassandra cluster, with both
> Linux and Windows nodes? I tried doing it and got
>
> Error in ThreadPoolExecutor
> java.lang.NullPointerException
>
> Not sure if this is due to the Linux/Windows mix or something else.
>
>
> Details below:
[...]
>
> INFO 21:42:08,091 Bootstrapping
>
> ERROR 21:49:03,526 Error in ThreadPoolExecutor
>
> java.lang.NullPointerException
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
>
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:619)
>
> ERROR 21:49:03,527 Fatal exception in thread
> Thread[MESSAGE-DESERIALIZER-POOL:1,5,main]
>
> java.lang.NullPointerException
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
>
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:619)
>
>
Looks like https://issues.apache.org/jira/browse/CASSANDRA-1136
Make sure you have enough nodes in your cluster to satisfy your
replication factor before you add any data. This is what seems to be
source of the problem in my case.
That said, I was also using an heterogeneous system (Linux + Windows)
but I think I tested it with only Linux nodes too.
Re: Heterogeneous Cassandra Cluster
Posted by Jonathan Ellis <jb...@gmail.com>.
No. And if we did it would be a bad idea: good ops practice is to
_minimize_ variability.
On Wed, Jun 2, 2010 at 3:18 AM, David Boxenhorn <da...@lookin2.com> wrote:
> Is it possible to make a heterogeneous Cassandra cluster, with both Linux
> and Windows nodes? I tried doing it and got
>
> Error in ThreadPoolExecutor
> java.lang.NullPointerException
>
> Not sure if this is due to the Linux/Windows mix or something else.
>
>
> Details below:
>
>
>
> [root@IQDEV01 cassandra]# bin/cassandra -f
>
> INFO 20:32:26,431 Auto DiskAccessMode determined to be mmap
>
> INFO 20:32:27,085 Sampling index for
> /var/lib/cassandra/data/system/LocationInfo-1-Data.db
>
> INFO 20:32:27,095 Sampling index for
> /var/lib/cassandra/data/system/LocationInfo-2-Data.db
>
> INFO 20:32:27,104 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1275412410865.log
>
> INFO 20:32:27,129 Creating new commitlog segment
> /var/lib/cassandra/commitlog/CommitLog-1275413547129.log
>
> INFO 20:32:27,138 LocationInfo has reached its threshold; switching in a
> fresh Memtable at
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1275413547129.log',
> position=173)
>
> INFO 20:32:27,138 Enqueuing flush of Memtable(LocationInfo)@1491010616
>
> INFO 20:32:27,139 Writing Memtable(LocationInfo)@1491010616
>
> INFO 20:32:27,187 Completed flushing
> /var/lib/cassandra/data/system/LocationInfo-3-Data.db
>
> INFO 20:32:27,207 Log replay complete
>
> INFO 20:32:27,239 Saved Token found: 25870423804996813139937576731363583348
>
> INFO 20:32:27,239 Saved ClusterName found: Lookin2
>
> INFO 20:32:27,247 Starting up server gossip
>
> INFO 20:32:27,266 Joining: getting load information
>
> INFO 20:32:27,267 Sleeping 90000 ms to wait for load information...
>
> INFO 20:32:27,327 Node /192.168.80.12 is now part of the cluster
>
> INFO 20:32:27,332 Node /192.168.80.234 is now part of the cluster
>
> INFO 20:32:27,864 InetAddress /192.168.80.12 is now UP
>
> INFO 20:32:27,872 InetAddress /192.168.80.234 is now UP
>
> INFO 20:33:57,269 Joining: getting bootstrap token
>
> INFO 20:33:57,278 New token will be 25870423804996813139937576731363583348
> to assume load from /192.168.80.12
>
> INFO 20:33:57,279 Joining: sleeping 30000 for pending range setup
>
> INFO 20:34:27,280 Bootstrapping
>
> INFO 21:32:27,867 Compacting []
>
> INFO 21:38:27,118 LocationInfo has reached its threshold; switching in a
> fresh Memtable at
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1275413547129.log',
> position=824)
>
> INFO 21:38:27,118 Enqueuing flush of Memtable(LocationInfo)@993374707
>
> INFO 21:38:27,118 Writing Memtable(LocationInfo)@993374707
>
> INFO 21:38:27,158 Completed flushing
> /var/lib/cassandra/data/system/LocationInfo-4-Data.db
>
> INFO 21:38:27,160 Compacting
> [org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-1-Data.db'),org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-2-Data.db'),org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-3-Data.db'),org.apache.cassandra.io.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-4-Data.db')]
>
> INFO 21:38:27,217 Compacted to
> /var/lib/cassandra/data/system/LocationInfo-5-Data.db. 1294/358 bytes for 1
> keys. Time: 56ms.
>
> [root@IQDEV01 cassandra]# bin/cassandra -f
>
> INFO 21:40:07,519 Auto DiskAccessMode determined to be mmap
>
> INFO 21:40:07,972 Deleted
> /var/lib/cassandra/data/system/LocationInfo-1-Data.db
>
> INFO 21:40:07,973 Deleted
> /var/lib/cassandra/data/system/LocationInfo-2-Data.db
>
> INFO 21:40:07,974 Deleted
> /var/lib/cassandra/data/system/LocationInfo-3-Data.db
>
> INFO 21:40:07,982 Sampling index for
> /var/lib/cassandra/data/system/LocationInfo-5-Data.db
>
> INFO 21:40:07,991 Deleted
> /var/lib/cassandra/data/system/LocationInfo-4-Data.db
>
> INFO 21:40:08,000 Replaying
> /var/lib/cassandra/commitlog/CommitLog-1275413547129.log
>
> INFO 21:40:08,001 Log replay complete
>
> INFO 21:40:08,038 Saved Token found: 25870423804996813139937576731363583348
>
> INFO 21:40:08,040 Saved ClusterName found: Lookin2
>
> INFO 21:40:08,042 Creating new commitlog segment
> /var/lib/cassandra/commitlog/CommitLog-1275417608042.log
>
> INFO 21:40:08,059 Starting up server gossip
>
> INFO 21:40:08,071 Joining: getting load information
>
> INFO 21:40:08,071 Sleeping 90000 ms to wait for load information...
>
> INFO 21:40:10,372 Node /192.168.80.12 is now part of the cluster
>
> INFO 21:40:10,374 Node /192.168.80.234 is now part of the cluster
>
> INFO 21:40:11,091 InetAddress /192.168.80.234 is now UP
>
> INFO 21:40:12,078 InetAddress /192.168.80.12 is now UP
>
> INFO 21:41:38,072 Joining: getting bootstrap token
>
> INFO 21:41:38,088 New token will be 25870423804996813139937576731363583348
> to assume load from /192.168.80.12
>
> INFO 21:41:38,089 Joining: sleeping 30000 for pending range setup
>
> INFO 21:42:08,091 Bootstrapping
>
> ERROR 21:49:03,526 Error in ThreadPoolExecutor
>
> java.lang.NullPointerException
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
>
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:619)
>
> ERROR 21:49:03,527 Fatal exception in thread
> Thread[MESSAGE-DESERIALIZER-POOL:1,5,main]
>
> java.lang.NullPointerException
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.getNewNames(StreamInitiateVerbHandler.java:154)
>
> at
> org.apache.cassandra.streaming.StreamInitiateVerbHandler.doVerb(StreamInitiateVerbHandler.java:76)
>
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:619)
>
> INFO 22:40:08,015 LocationInfo has reached its threshold; switching in a
> fresh Memtable at
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1275417608042.log',
> position=824)
>
> INFO 22:40:08,016 Enqueuing flush of Memtable(LocationInfo)@925668695
>
> INFO 22:40:08,017 Writing Memtable(LocationInfo)@925668695
>
> INFO 22:40:08,058 Completed flushing
> /var/lib/cassandra/data/system/LocationInfo-6-Data.db
>
> INFO 22:40:11,100 Compacting []
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com