You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Alan Burlison <Al...@sun.com> on 2009/07/23 23:34:45 UTC
Derby replication failed under heavy load
Anyone seen this before?
> Replication slave role was stopped for database 'opensolaris'.
>
> ------------ BEGIN SHUTDOWN ERROR STACK -------------
>
> ERROR XSLA7: Cannot redo operation null in the log.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.store.raw.log.FileLogger.redo(Unknown Source)
> at org.apache.derby.impl.store.raw.log.LogToFile.recover(Unknown Source)
> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
> at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
> at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
> at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
> at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
> at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
> at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(Unknown Source)
> at org.apache.derby.impl.db.SlaveDatabase.access$000(Unknown Source)
> at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: ERROR 08006: Database '{0}' shutdown.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.store.raw.log.LogToFile.stopReplicationSlaveRole(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.stopSlave(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.access$500(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController$SlaveLogReceiverThread.run(Unknown Source)
> ============= begin nested exception, level (1) ===========
> ERROR 08006: Database '{0}' shutdown.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.store.raw.log.LogToFile.stopReplicationSlaveRole(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.stopSlave(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.access$500(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController$SlaveLogReceiverThread.run(Unknown Source)
> ============= end nested exception, level (1) ===========
>
>
> ------------ END SHUTDOWN ERROR STACK -------------
>
> Replication slave role was stopped for database 'opensolaris'.
>
> 2009-07-23 21:24:08.530 GMT:
> Shutting down instance 601a400f-0122-a88d-61ad-000031b5248d
> ----------------------------------------------------------------
--
Alan Burlison
--
Re: Derby replication failed under heavy load
Posted by Alan Burlison <Al...@sun.com>.
Knut Anders Hatlen wrote:
>> What I don't understand is why the master isn't being throttled if the
>> slave can't keep up.
>
> That sounds like DERBY-3719, which Dag fixed for the upcoming 10.5.2
> release.
Yay for Dag! :-)
We can live with this until the fix arrives, because we are only doing
this particular operation during the initial bulk upload to the
database, in normal operation we don't see it.
--
Alan Burlison
--
Re: Derby replication failed under heavy load
Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Alan Burlison <Al...@Sun.COM> writes:
> Andrew Lawrenson wrote:
>
>> I'm not sure if it's the exact same issue, but I have observed the
>> replication failing under heavy load where the replication buffers
>> would fill faster than the logs can be shipped to the slave (even
>> with fast servers & fast network). I've previously tweaked a copy of
>> derby to allow up to 100 buffers (rather than the current max of 10),
>> and optimized the various thresholds, but still observed over 80
>> buffers being used under high load (this was with 10.4.2.0 on
>> Solaris).
>
> I think it is the same issue - I saw an errors in the master log that
> mentioned something about the buffers filling up. The thing that
> seems to trigger it is doing a number of whole-table deletes,
> presumably because while that's being processed on the slave the
> master is still chugging along writing more log records.
>
> What I don't understand is why the master isn't being throttled if the
> slave can't keep up.
That sounds like DERBY-3719, which Dag fixed for the upcoming 10.5.2
release.
--
Knut Anders
Re: Derby replication failed under heavy load
Posted by Alan Burlison <Al...@sun.com>.
Andrew Lawrenson wrote:
> I'm not sure if it's the exact same issue, but I have observed the
> replication failing under heavy load where the replication buffers
> would fill faster than the logs can be shipped to the slave (even
> with fast servers & fast network). I've previously tweaked a copy of
> derby to allow up to 100 buffers (rather than the current max of 10),
> and optimized the various thresholds, but still observed over 80
> buffers being used under high load (this was with 10.4.2.0 on
> Solaris).
I think it is the same issue - I saw an errors in the master log that
mentioned something about the buffers filling up. The thing that seems
to trigger it is doing a number of whole-table deletes, presumably
because while that's being processed on the slave the master is still
chugging along writing more log records.
What I don't understand is why the master isn't being throttled if the
slave can't keep up.
--
Alan Burlison
--
RE: Derby replication failed under heavy load
Posted by Andrew Lawrenson <an...@coppereye.com>.
I'm not sure if it's the exact same issue, but I have observed the replication failing under heavy load where the replication buffers would fill faster than the logs can be shipped to the slave (even with fast servers & fast network).
I've previously tweaked a copy of derby to allow up to 100 buffers (rather than the current max of 10), and optimized the various thresholds, but still observed over 80 buffers being used under high load (this was with 10.4.2.0 on Solaris).
Andy
-----Original Message-----
From: Alan Burlison [mailto:Alan.Burlison@sun.com]
Sent: 23 July 2009 22:35
To: Derby Discussion
Subject: Derby replication failed under heavy load
Anyone seen this before?
> Replication slave role was stopped for database 'opensolaris'.
>
> ------------ BEGIN SHUTDOWN ERROR STACK -------------
>
> ERROR XSLA7: Cannot redo operation null in the log.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.store.raw.log.FileLogger.redo(Unknown Source)
> at org.apache.derby.impl.store.raw.log.LogToFile.recover(Unknown Source)
> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
> at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
> at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
> at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
> at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
> at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
> at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(Unknown Source)
> at org.apache.derby.impl.db.SlaveDatabase.access$000(Unknown Source)
> at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: ERROR 08006: Database '{0}' shutdown.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.store.raw.log.LogToFile.stopReplicationSlaveRole(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.stopSlave(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.access$500(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController$SlaveLogReceiverThread.run(Unknown Source)
> ============= begin nested exception, level (1) ===========
> ERROR 08006: Database '{0}' shutdown.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.store.raw.log.LogToFile.stopReplicationSlaveRole(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.stopSlave(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController.access$500(Unknown Source)
> at org.apache.derby.impl.store.replication.slave.SlaveController$SlaveLogReceiverThread.run(Unknown Source)
> ============= end nested exception, level (1) ===========
>
>
> ------------ END SHUTDOWN ERROR STACK -------------
>
> Replication slave role was stopped for database 'opensolaris'.
>
> 2009-07-23 21:24:08.530 GMT:
> Shutting down instance 601a400f-0122-a88d-61ad-000031b5248d
> ----------------------------------------------------------------
--
Alan Burlison
--