You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by jlindwall <jl...@yahoo.com> on 2014/02/11 21:21:53 UTC
Leveldb on Solaris?
I'm trying out leveldb persistence on Solaris (sparc). Is this a supported
platform?
I get a DEBUG msg/stacktrace at startup saying the system cannot find
leveldbjni library, but I assume that is ok since we are using the pure java
implementation.
I then run my test with 100 durable subscribers and 50 publishers runnning
at full speed and after abut 3000-4000 messages the leveldb database seems
to get corrupted and the broker stops accepting connections. I tried using
a non-NFS disk for the db but it made no difference.
Here's the log output; kinda weird that a condition that is shutting down
the broker is at level INFO:
2014-02-11 12:14:50,819 | INFO | Stopping BrokerService[localhost] due to
exception, java.io.IOException:
com.google.common.util.concurrent.UncheckedExecutionException:
org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
at 22 | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB
IOException handler.
java.io.IOException:
com.google.common.util.concurrent.UncheckedExecutionException:
org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
at 22
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:543)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:974)
at
org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1302)
at
org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:600)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
com.google.common.util.concurrent.UncheckedExecutionException:
org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
at 22
at
org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
at org.iq80.leveldb.impl.DbImpl.createWriteBatch(DbImpl.java:707)
at
org.apache.activemq.leveldb.LevelDBClient$RichDB.write(LevelDBClient.scala:220)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1$$anonfun$apply$mcV$sp$14.apply(LevelDBClient.scala:1304)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1$$anonfun$apply$mcV$sp$14.apply(LevelDBClient.scala:1303)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$appender$1.apply(RecordLog.scala:475)
at
org.apache.activemq.leveldb.util.TimeMetric.apply(TimeMetric.scala:43)
at
org.apache.activemq.leveldb.RecordLog.appender(RecordLog.scala:474)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:1303)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:1302)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:1302)
at
org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.scala:968)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_index$1.apply(LevelDBClient.scala:974)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:540)
... 7 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
at 22
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2256)
at com.google.common.cache.LocalCache.get(LocalCache.java:3980)
at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3984)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4868)
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:80)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:69)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:64)
at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:983)
at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:932)
at
org.iq80.leveldb.impl.DbImpl.compactMemTableInternal(DbImpl.java:896)
at
org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:455)
at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
Caused by: org.iq80.snappy.CorruptionException: Invalid copy offset for
opcode starting at 22
at
org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
at
org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:76)
at org.iq80.snappy.Snappy.uncompress(Snappy.java:43)
at
org.iq80.leveldb.util.Snappy$IQ80Snappy.uncompress(Snappy.java:100)
at org.iq80.leveldb.util.Snappy.uncompress(Snappy.java:160)
at
org.iq80.leveldb.table.FileChannelTable.readBlock(FileChannelTable.java:74)
at org.iq80.leveldb.table.Table.<init>(Table.java:60)
at
org.iq80.leveldb.table.FileChannelTable.<init>(FileChannelTable.java:34)
at
org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:117)
at
org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:102)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:57)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:54)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3579)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
... 18 more
2014-02-11 12:14:50,827 | INFO | Apache ActiveMQ 5.9.0 (localhost,
ID:testapp01.xifin.com-51509-1392149584977-0:1) is shutting down |
org.apache.activemq.broker.BrokerService | IOExceptionHandler: stopping
BrokerService[localhost]
2014-02-11 12:14:50,831 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire
| org.apache.activemq.broker.jmx.ManagementContext | IOExceptionHandler:
stopping BrokerService[localhost]
--
View this message in context: http://activemq.2283324.n4.nabble.com/Leveldb-on-Solaris-tp4677824.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Leveldb on Solaris?
Posted by jlindwall <jl...@yahoo.com>.
I have reported this issue in jira:
https://issues.apache.org/jira/browse/AMQ-5900
I'm surprised at my struggle to identify a reliable HA solution for
activemq. Not sure if the issue is Solaris or issues in my config or
something else.
1. master/slave fail-over was a bust due to
https://issues.apache.org/jira/browse/AMQ-5897
2. leveldb replication fails as described above
3. JDBC persistence seems to be my only option. It appears to work properly
in the face of a master failure at least. I need to measure performance now
to see if it will be acceptable.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Leveldb-on-Solaris-tp4677824p4699767.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Leveldb on Solaris?
Posted by jlindwall <jl...@yahoo.com>.
There still seems to be leveldb issues on Solaris under activemq 5.11.1. I'm
using Solaris 5.11. I had 3 activemq nodes running using leveldb
replication. I connected a client that listened for messages, but I did not
send any messages at all.
I then used "kil -9" to kill the master node. The client failed to
reconnect even htough I used a failover url. In the node2 activemq.log I
see the following, as it attempted to become the new master:
2015-07-22 17:57:19,334 | INFO | Attaching to master:
tcp://172.10.10.10:61619 |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2015-07-22 17:57:19,338 | WARN | Unexpected session error:
java.net.ConnectException: Connection refused |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2015-07-22 17:57:20,044 | INFO | Not enough cluster members have reported
their update positions yet. |
org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
2015-07-22 17:57:20,059 | INFO | Slave stopped |
org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[xifin] Task-3
2015-07-22 17:57:20,061 | INFO | Not enough cluster members have reported
their update positions yet. |
org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[xifin] Task-3
2015-07-22 17:57:20,068 | INFO | Not enough cluster members have reported
their update positions yet. |
org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
2015-07-22 17:57:20,087 | INFO | Promoted to master |
org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
2015-07-22 17:57:20,124 | INFO | Using the pure java LevelDB
implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
BrokerService[xifin] Task-3
2015-07-22 17:57:20,380 | INFO | No IOExceptionHandler registered, ignoring
IO exception | org.apache.activemq.broker.BrokerService | LevelDB
IOException handler.
java.io.IOException: org.iq80.snappy.CorruptionException: Invalid copy
offset for opcode starting at 8
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:667)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:110)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.11.1.jar:5.11.1]
at
org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:230)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_60]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_60]
2015-07-22 17:57:20,400 | INFO | Stopped
LevelDB[/home/jlindwall/servers/activemq-replicated-leveldb-cluster/node2/data/LevelDB]
| org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.
Anybody successfully using leveldb on solaris?
Thanks!
--
View this message in context: http://activemq.2283324.n4.nabble.com/Leveldb-on-Solaris-tp4677824p4699723.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Leveldb on Solaris?
Posted by Timothy Bish <ta...@gmail.com>.
On 02/11/2014 03:40 PM, jlindwall wrote:
> I'll try that snapshot, thanks. Um ... my svn skills are rusty. ANy idea why
> this "svn co" command below does not actually seem to do anything?
>
> $ svn checkout
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/
> svn: OPTIONS of
> 'https://repository.apache.org/content/repositories/snapshots/org/apache/activemq':
> 200 OK (https://repository.apache.org)
>
> Thanks!
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Leveldb-on-Solaris-tp4677824p4677826.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
See:
http://activemq.apache.org/source.html
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
Re: Leveldb on Solaris?
Posted by jlindwall <jl...@yahoo.com>.
I'll try that snapshot, thanks. Um ... my svn skills are rusty. ANy idea why
this "svn co" command below does not actually seem to do anything?
$ svn checkout
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/
svn: OPTIONS of
'https://repository.apache.org/content/repositories/snapshots/org/apache/activemq':
200 OK (https://repository.apache.org)
Thanks!
--
View this message in context: http://activemq.2283324.n4.nabble.com/Leveldb-on-Solaris-tp4677824p4677826.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Leveldb on Solaris?
Posted by Timothy Bish <ta...@gmail.com>.
There was a lot of fixing done for LevelDB after v5.9.0 was released.
You might want to try the same test using a 5.10-SNAPSHOT and see how
that goes.
On 02/11/2014 03:21 PM, jlindwall wrote:
> I'm trying out leveldb persistence on Solaris (sparc). Is this a supported
> platform?
>
> I get a DEBUG msg/stacktrace at startup saying the system cannot find
> leveldbjni library, but I assume that is ok since we are using the pure java
> implementation.
>
> I then run my test with 100 durable subscribers and 50 publishers runnning
> at full speed and after abut 3000-4000 messages the leveldb database seems
> to get corrupted and the broker stops accepting connections. I tried using
> a non-NFS disk for the db but it made no difference.
>
> Here's the log output; kinda weird that a condition that is shutting down
> the broker is at level INFO:
>
> 2014-02-11 12:14:50,819 | INFO | Stopping BrokerService[localhost] due to
> exception, java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
> at 22 | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB
> IOException handler.
> java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
> at 22
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:543)
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:974)
> at
> org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1302)
> at
> org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:600)
> at
> org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
> at 22
> at
> org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
> at org.iq80.leveldb.impl.DbImpl.createWriteBatch(DbImpl.java:707)
> at
> org.apache.activemq.leveldb.LevelDBClient$RichDB.write(LevelDBClient.scala:220)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1$$anonfun$apply$mcV$sp$14.apply(LevelDBClient.scala:1304)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1$$anonfun$apply$mcV$sp$14.apply(LevelDBClient.scala:1303)
> at
> org.apache.activemq.leveldb.RecordLog$$anonfun$appender$1.apply(RecordLog.scala:475)
> at
> org.apache.activemq.leveldb.util.TimeMetric.apply(TimeMetric.scala:43)
> at
> org.apache.activemq.leveldb.RecordLog.appender(RecordLog.scala:474)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:1303)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:1302)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:1302)
> at
> org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.scala:968)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_index$1.apply(LevelDBClient.scala:974)
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:540)
> ... 7 more
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting
> at 22
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2256)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3980)
> at
> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3984)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4868)
> at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:80)
> at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:69)
> at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:64)
> at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:983)
> at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:932)
> at
> org.iq80.leveldb.impl.DbImpl.compactMemTableInternal(DbImpl.java:896)
> at
> org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:455)
> at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
> at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
> at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
> at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> Caused by: org.iq80.snappy.CorruptionException: Invalid copy offset for
> opcode starting at 22
> at
> org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
> at
> org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:76)
> at org.iq80.snappy.Snappy.uncompress(Snappy.java:43)
> at
> org.iq80.leveldb.util.Snappy$IQ80Snappy.uncompress(Snappy.java:100)
> at org.iq80.leveldb.util.Snappy.uncompress(Snappy.java:160)
> at
> org.iq80.leveldb.table.FileChannelTable.readBlock(FileChannelTable.java:74)
> at org.iq80.leveldb.table.Table.<init>(Table.java:60)
> at
> org.iq80.leveldb.table.FileChannelTable.<init>(FileChannelTable.java:34)
> at
> org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:117)
> at
> org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:102)
> at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:57)
> at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:54)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3579)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
> ... 18 more
> 2014-02-11 12:14:50,827 | INFO | Apache ActiveMQ 5.9.0 (localhost,
> ID:testapp01.xifin.com-51509-1392149584977-0:1) is shutting down |
> org.apache.activemq.broker.BrokerService | IOExceptionHandler: stopping
> BrokerService[localhost]
> 2014-02-11 12:14:50,831 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire
> | org.apache.activemq.broker.jmx.ManagementContext | IOExceptionHandler:
> stopping BrokerService[localhost]
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Leveldb-on-Solaris-tp4677824.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/