You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Christian Grassi <ch...@gmail.com> on 2014/12/16 16:39:11 UTC

ActiveMQ 5.10 LevelDB Issues

I have a 3 node 5.10 cluster with replicated levelDB as persistent store.


<persistenceAdapter>
         <replicatedLevelDB
          directory="activemq-data"
          replicas="3"
          bind="tcp://0.0.0.0:0"
          zkAddress="queue1:2181,queue2:2181,queue3:2181"
          zkPath="/activemq/leveldb-stores"
          hostname="queue3"
          sync="quorum_disk"
             />
        </persistenceAdapter>


I'm Stresstesting the queue with around 5000 persistent msg/s.
1 Producer
2 Consumers

I get some warning messages on the log:
On Master:
2014-12-16 16:25:28,375 | INFO  | Slave has disconnected:
db6c9a23-7025-4384-b02f-dcda763113c3 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-3
2014-12-16 16:25:28,761 | WARN  | Unexpected session error:
java.io.FileNotFoundException:
/queue/activemq/conf/activemq-data/00000027b8d097b0.log (No such file or
directory) | org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-1
2014-12-16 16:25:28,761 | INFO  | Slave has disconnected:
6a84579c-77fe-41eb-a728-1be472c12894 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-2
2014-12-16 16:25:29,603 | INFO  | Slave has connected:
db6c9a23-7025-4384-b02f-dcda763113c3 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-2
2014-12-16 16:25:30,034 | INFO  | Slave has connected:
6a84579c-77fe-41eb-a728-1be472c12894 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-1
2014-12-16 16:25:31,360 | INFO  | Slave has now caught up:
6a84579c-77fe-41eb-a728-1be472c12894 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-4
2014-12-16 16:25:32,546 | INFO  | Slave has now caught up:
db6c9a23-7025-4384-b02f-dcda763113c3 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-2


On the slaves

2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN  | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882






It looks to me that from time to time the connection to the master drops.

Should I worry about it ? what No reader available means, is there data
loss risk?

It looks to me that the level db implementation is kind of fragile...

Is somebody using it in production ?

Regards


Christian