You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roshan <co...@gmail.com> on 2012/03/13 03:31:12 UTC
Several times hinted handoff for the same node with Cassandra 1.0.8
Hi
I have upgrade our development Cassandra cluster (2 nodes) from 1.0.6 to
1.0.8 version.
After upgrade to 1.0.8 version, one node keep trying to send hints every 10
minutes (seems). There is no any network issues between two nodes and can do
ping with using server name and ip address. Also the nodetool ring is
working fine from both end. Could someone please help on this?
Here is the Cassandra stack of the problem node:
2012-03-13 13:12:56,138 INFO [StorageService] Cassandra version: 1.0.8
2012-03-13 13:12:56,138 INFO [StorageService] Thrift API version: 19.20.0
2012-03-13 13:12:56,138 INFO [StorageService] Loading persisted ring state
2012-03-13 13:12:56,149 INFO [StorageService] Starting up server gossip
2012-03-13 13:12:56,150 INFO [ColumnFamilyStore] Enqueuing flush of
Memtable-LocationInfo@1899843147(29/36 serialized/live bytes, 1 ops)
2012-03-13 13:12:56,152 INFO [Memtable] Writing
Memtable-LocationInfo@1899843147(29/36 serialized/live bytes, 1 ops)
2012-03-13 13:12:56,238 INFO [Memtable] Completed flushing
/data/cassandradb/data/system/LocationInfo-hc-50-Data.db (80 bytes)
2012-03-13 13:12:56,253 INFO [MessagingService] Starting Messaging Service
on port 7000
2012-03-13 13:12:56,259 INFO [StorageService] Using saved token
144939581272443147669723010154540982565
2012-03-13 13:12:56,261 INFO [ColumnFamilyStore] Enqueuing flush of
Memtable-LocationInfo@1915784320(53/66 serialized/live bytes, 2 ops)
2012-03-13 13:12:56,261 INFO [Memtable] Writing
Memtable-LocationInfo@1915784320(53/66 serialized/live bytes, 2 ops)
2012-03-13 13:12:56,350 INFO [Gossiper] Node /10.1.161.67 has restarted,
now UP
2012-03-13 13:12:56,350 INFO [Gossiper] InetAddress /10.1.161.67 is now UP
2012-03-13 13:12:56,351 INFO [StorageService] Node /10.1.161.67 state jump
to normal
2012-03-13 13:12:56,378 INFO [Memtable] Completed flushing
/data/cassandradb/data/system/LocationInfo-hc-51-Data.db (163 bytes)
2012-03-13 13:12:56,389 INFO [CompactionTask] Compacting
[SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-51-Data.db'),
SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-50-Data.db'),
SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-48-Data.db'),
SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-49-Data.db')]
2012-03-13 13:12:56,402 INFO [StorageService] Node
app8.dev1.net/10.1.161.68 state jump to normal
2012-03-13 13:12:56,403 INFO [StorageService] Bootstrap/Replace/Move
completed! Now serving reads.
2012-03-13 13:12:56,403 INFO [Mx4jTool] Will not load MX4J, mx4j-tools.jar
is not in the classpath
2012-03-13 13:12:56,438 INFO [CassandraDaemon] Binding thrift service to
app8.dev1.net/10.1.161.68:9160
2012-03-13 13:12:56,445 INFO [CassandraDaemon] Using TFastFramedTransport
with a max frame size of 62914560 bytes.
2012-03-13 13:12:56,448 INFO [CassandraDaemon] Using synchronous/threadpool
thrift server on app8.dev1.net/10.1.161.68 : 9160
2012-03-13 13:12:56,448 INFO [CassandraDaemon] Listening for thrift
clients...
2012-03-13 13:12:56,498 INFO [CompactionTask] Compacted to
[/data/cassandradb/data/system/LocationInfo-hc-52-Data.db,]. 844 to 438
(~51% of original) bytes for 4 keys at 0.003941MB/s. Time: 106ms.
2012-03-13 13:13:42,410 INFO [HintedHandOffManager] Started hinted handoff
for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
2012-03-13 13:13:42,490 INFO [HintedHandOffManager] Finished hinted handoff
of 0 rows to endpoint /10.1.161.67
2012-03-13 13:23:28,946 INFO [HintedHandOffManager] Started hinted handoff
for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
2012-03-13 13:23:28,948 INFO [HintedHandOffManager] Finished hinted handoff
of 0 rows to endpoint /10.1.161.67
Waiting for data... (interrupt to abort)
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Several-times-hinted-handoff-for-the-same-node-with-Cassandra-1-0-8-tp7367386p7367386.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Several times hinted handoff for the same node with Cassandra 1.0.8
Posted by Brandon Williams <dr...@gmail.com>.
Just ignore it: https://issues.apache.org/jira/browse/CASSANDRA-3955
On Mon, Mar 12, 2012 at 9:31 PM, Roshan <co...@gmail.com> wrote:
> Hi
>
> I have upgrade our development Cassandra cluster (2 nodes) from 1.0.6 to
> 1.0.8 version.
>
> After upgrade to 1.0.8 version, one node keep trying to send hints every 10
> minutes (seems). There is no any network issues between two nodes and can do
> ping with using server name and ip address. Also the nodetool ring is
> working fine from both end. Could someone please help on this?
>
> Here is the Cassandra stack of the problem node:
>
> 2012-03-13 13:12:56,138 INFO [StorageService] Cassandra version: 1.0.8
> 2012-03-13 13:12:56,138 INFO [StorageService] Thrift API version: 19.20.0
> 2012-03-13 13:12:56,138 INFO [StorageService] Loading persisted ring state
> 2012-03-13 13:12:56,149 INFO [StorageService] Starting up server gossip
> 2012-03-13 13:12:56,150 INFO [ColumnFamilyStore] Enqueuing flush of
> Memtable-LocationInfo@1899843147(29/36 serialized/live bytes, 1 ops)
> 2012-03-13 13:12:56,152 INFO [Memtable] Writing
> Memtable-LocationInfo@1899843147(29/36 serialized/live bytes, 1 ops)
> 2012-03-13 13:12:56,238 INFO [Memtable] Completed flushing
> /data/cassandradb/data/system/LocationInfo-hc-50-Data.db (80 bytes)
> 2012-03-13 13:12:56,253 INFO [MessagingService] Starting Messaging Service
> on port 7000
> 2012-03-13 13:12:56,259 INFO [StorageService] Using saved token
> 144939581272443147669723010154540982565
> 2012-03-13 13:12:56,261 INFO [ColumnFamilyStore] Enqueuing flush of
> Memtable-LocationInfo@1915784320(53/66 serialized/live bytes, 2 ops)
> 2012-03-13 13:12:56,261 INFO [Memtable] Writing
> Memtable-LocationInfo@1915784320(53/66 serialized/live bytes, 2 ops)
> 2012-03-13 13:12:56,350 INFO [Gossiper] Node /10.1.161.67 has restarted,
> now UP
> 2012-03-13 13:12:56,350 INFO [Gossiper] InetAddress /10.1.161.67 is now UP
> 2012-03-13 13:12:56,351 INFO [StorageService] Node /10.1.161.67 state jump
> to normal
> 2012-03-13 13:12:56,378 INFO [Memtable] Completed flushing
> /data/cassandradb/data/system/LocationInfo-hc-51-Data.db (163 bytes)
> 2012-03-13 13:12:56,389 INFO [CompactionTask] Compacting
> [SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-51-Data.db'),
> SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-50-Data.db'),
> SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-48-Data.db'),
> SSTableReader(path='/data/cassandradb/data/system/LocationInfo-hc-49-Data.db')]
> 2012-03-13 13:12:56,402 INFO [StorageService] Node
> app8.dev1.net/10.1.161.68 state jump to normal
> 2012-03-13 13:12:56,403 INFO [StorageService] Bootstrap/Replace/Move
> completed! Now serving reads.
> 2012-03-13 13:12:56,403 INFO [Mx4jTool] Will not load MX4J, mx4j-tools.jar
> is not in the classpath
> 2012-03-13 13:12:56,438 INFO [CassandraDaemon] Binding thrift service to
> app8.dev1.net/10.1.161.68:9160
> 2012-03-13 13:12:56,445 INFO [CassandraDaemon] Using TFastFramedTransport
> with a max frame size of 62914560 bytes.
> 2012-03-13 13:12:56,448 INFO [CassandraDaemon] Using synchronous/threadpool
> thrift server on app8.dev1.net/10.1.161.68 : 9160
> 2012-03-13 13:12:56,448 INFO [CassandraDaemon] Listening for thrift
> clients...
> 2012-03-13 13:12:56,498 INFO [CompactionTask] Compacted to
> [/data/cassandradb/data/system/LocationInfo-hc-52-Data.db,]. 844 to 438
> (~51% of original) bytes for 4 keys at 0.003941MB/s. Time: 106ms.
> 2012-03-13 13:13:42,410 INFO [HintedHandOffManager] Started hinted handoff
> for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
> 2012-03-13 13:13:42,490 INFO [HintedHandOffManager] Finished hinted handoff
> of 0 rows to endpoint /10.1.161.67
> 2012-03-13 13:23:28,946 INFO [HintedHandOffManager] Started hinted handoff
> for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
> 2012-03-13 13:23:28,948 INFO [HintedHandOffManager] Finished hinted handoff
> of 0 rows to endpoint /10.1.161.67
> Waiting for data... (interrupt to abort)
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Several-times-hinted-handoff-for-the-same-node-with-Cassandra-1-0-8-tp7367386p7367386.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.