You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-commits@db.apache.org by mi...@apache.org on 2006/02/07 13:08:08 UTC
svn commit: r375583 - in /db/derby/code/trunk/java:
engine/org/apache/derby/iapi/reference/
engine/org/apache/derby/iapi/services/cache/
engine/org/apache/derby/impl/services/cache/
engine/org/apache/derby/impl/store/raw/data/ engine/org/apache/derby/i...
Author: mikem
Date: Tue Feb 7 04:07:14 2006
New Revision: 375583
URL: http://svn.apache.org/viewcvs?rev=375583&view=rev
Log:
DERBY-888
initial partial checkin of allocation improvement. This checkin changes
the default of the system to not sync on allocation. All tests pass,
and hand testing of allocation encountering bad pages during redo worked.
Also adds back in development only path to be able to turn on and off
various sync options under an insane build, to compare affect of log sync,
vs. allocation sync, vs. checkpoint sync.
Modified:
db/derby/code/trunk/java/engine/org/apache/derby/iapi/reference/Property.java
db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/cache/CacheManager.java
db/derby/code/trunk/java/engine/org/apache/derby/impl/services/cache/Clock.java
db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java
db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
db/derby/code/trunk/java/engine/org/apache/derby/loc/messages_en.properties
db/derby/code/trunk/java/shared/org/apache/derby/shared/common/reference/SQLState.java
Modified: db/derby/code/trunk/java/engine/org/apache/derby/iapi/reference/Property.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/iapi/reference/Property.java?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/iapi/reference/Property.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/iapi/reference/Property.java Tue Feb 7 04:07:14 2006
@@ -740,4 +740,52 @@
public static final String SERVICE_PROTOCOL = "derby.serviceProtocol";
public static final String SERVICE_LOCALE = "derby.serviceLocale";
+ /**
+ * db2j.storage.dataNotSyncedAtCheckPoint
+ * <p>
+ * When set, the store system will not force a sync() call on the
+ * containers during a checkpoint.
+ * <p>
+ * An internal debug system only flag. The recovery system will not
+ * work properly if this flag is enabled, it is provided to do performance
+ * debugging to see whether the system is I/O bound based on checkpoint
+ * synchronous I/O.
+ * <p>
+ *
+ **/
+ public static final String STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT =
+ "db2j.storage.dataNotSyncedAtCheckPoint";
+
+ /**
+ * db2j.storage.dataNotSyncedAtAllocation
+ * <p>
+ * When set, the store system will not force a sync() call on the
+ * containers when pages are allocated.
+ * <p>
+ * An internal debug system only flag. The recovery system will not
+ * work properly if this flag is enabled, it is provided to do performance
+ * debugging to see whether the system is I/O bound based on page allocation
+ * synchronous I/O.
+ * <p>
+ *
+ **/
+ public static final String STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION =
+ "db2j.storage.dataNotSyncedAtAllocation";
+
+ /**
+ * db2j.storage.logNotSynced
+ * <p>
+ * When set, the store system will not force a sync() call on the log at
+ * commit.
+ * <p>
+ * An internal debug system only flag. The recovery system will not
+ * work properly if this flag is enabled, it is provided to do performance
+ * debugging to see whether the system is I/O bound based on log file
+ * synchronous I/O.
+ * <p>
+ *
+ **/
+ public static final String STORAGE_LOG_NOT_SYNCED =
+ "db2j.storage.logNotSynced";
+
}
Modified: db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/cache/CacheManager.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/cache/CacheManager.java?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/cache/CacheManager.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/cache/CacheManager.java Tue Feb 7 04:07:14 2006
@@ -199,10 +199,10 @@
/**
Place all objects in their clean state by calling their clean method
if they are dirty. This method guarantees that all objects that existed
- in the cache at the time of the call are placed in the clean state sometime
- during this call. Objects that are added to the cache during this call or
- objects that are dirtied during this call (by other callers) are not guaranteed
- to be clean once this call returns.
+ in the cache at the time of the call are placed in the clean state
+ sometime during this call. Objects that are added to the cache during
+ this call or objects that are dirtied during this call (by other
+ callers) are not guaranteed to be clean once this call returns.
@see Cacheable#clean
@see Cacheable#isDirty
@@ -216,9 +216,9 @@
Any cached object that results in the partialKey.equals(Object)
method returning true when passed the cached object will be cleaned.
<P>
- In order to clean more than one object the Cacheable equals method must be able to handle
- a partial key, e.g. a page has PageKey but a clean may pass a ContainerKey which will discard
- all pages in that container.
+ In order to clean more than one object the Cacheable equals method must
+ be able to handle a partial key, e.g. a page has PageKey but a clean
+ may pass a ContainerKey which will discard all pages in that container.
@exception StandardException Standard Cloudscape error policy.
*/
Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/services/cache/Clock.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/impl/services/cache/Clock.java?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/services/cache/Clock.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/services/cache/Clock.java Tue Feb 7 04:07:14 2006
@@ -1340,7 +1340,7 @@
// go from position (the last cached item in the holder array
// to 0 (the first). Otherwise, if we go from 0 to
// position, some other thread may come in and shrink items
- // which is between 0 and position. Since a shrink move all
+ // which are between 0 and position. Since a shrink moves all
// items up, we may skip some items without cleaning.
for ( ; position >= 0; position--, item = null) {
Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java Tue Feb 7 04:07:14 2006
@@ -180,8 +180,9 @@
private static final String LINE =
"----------------------------------------------------------------";
- // disable syncing of data during page allocation.
- boolean dataNotSyncedAtAllocation = false;
+ // disable syncing of data during page allocation. DERBY-888 changes
+ // the system to not require data syncing at allocation.
+ boolean dataNotSyncedAtAllocation = true;
// disable syncing of data during checkpoint.
boolean dataNotSyncedAtCheckpoint = false;
@@ -420,9 +421,6 @@
// - disable syncing of data during checkpoint.
dataNotSyncedAtCheckpoint = true;
- // - disable syncing of data during page allocation.
- dataNotSyncedAtAllocation = true;
-
// log message stating that derby.system.durability
// is set to a mode, where syncs wont be forced and the
// possible consequences of setting this mode
@@ -431,6 +429,25 @@
Property.DURABILITY_PROPERTY,
Property.DURABILITY_TESTMODE_NO_SYNC));
}
+ else if (Performance.MEASURE)
+ {
+ // development build only feature, must by hand set the
+ // Performance.MEASURE variable and rebuild. Useful during
+ // development to compare/contrast effect of syncing, release
+ // users can use the above relaxed durability option to disable
+ // all syncing.
+
+ // debug only flag - disable syncing of data during checkpoint.
+ dataNotSyncedAtCheckpoint =
+ PropertyUtil.getSystemBoolean(
+ Property.STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT);
+
+ if (dataNotSyncedAtCheckpoint)
+ Monitor.logMessage(
+ "Warning: " +
+ Property.STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT +
+ "set to true.");
+ }
fileHandler = new RFResource( this);
} // end of boot
@@ -1136,6 +1153,43 @@
}
+ /**
+ * Implement checkpoint operation, write/sync all pages in cache.
+ * <p>
+ * The derby write ahead log algorithm uses checkpoint of the data
+ * cache to determine points of the log no longer required by
+ * restart recovery.
+ * <p>
+ * This implementation uses the 2 cache interfaces to force all dirty
+ * pages to disk:
+ *
+ * WRITE DIRTY PAGES TO OS:
+ * In the first step all pages in the page cache
+ * are written, but not synced (pagecache.cleanAll). The cachemanager
+ * cleanAll() interface guarantees that every dirty page that exists
+ * when this call is first made will have it's clean() method called.
+ * The data cache (CachedPage.clean()), will call writePage but not
+ * sync the page.
+ * By using the java write then sync, the checkpoint is
+ * usually doing async I/O, allowing the OS to schedule multiple I/O's
+ * to the file as efficiently as it can.
+ * Note that it has been observed that checkpoints
+ * can flood the I/O system because these writes are not synced, see
+ * DERBY-799 - checkpoint should probably somehow restrict the rate
+ * it sends out those I/O's - it was observed a simple sleep every
+ * N writes fixed most of the problem.
+ *
+ * FORCE THOSE DIRTY WRITES TO DISK:
+ * To force the I/O's to disk, the system calls each open dirty file
+ * and uses the java interface to sync any outstanding dirty pages to
+ * disk (containerCache.cleanAll()). The open container cache does
+ * this work in RAFContainer.clean() by writing it's header out and
+ * syncing the file. (Note if any change is made to checkpoint to
+ * sync the writes vs. syncing the file, one probably still needs to
+ * write the container header out and sync it).
+ *
+ * @exception StandardException Standard exception policy.
+ **/
public void checkpoint() throws StandardException
{
pageCache.cleanAll();
Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java Tue Feb 7 04:07:14 2006
@@ -2877,6 +2877,24 @@
//if log not being synced;files shouldn't be open in write sync mode
isWriteSynced = false;
}
+ else if (Performance.MEASURE)
+ {
+ // development build only feature, must by hand set the
+ // Performance.MEASURE variable and rebuild. Useful during
+ // development to compare/contrast effect of syncing, release
+ // users can use the above relaxed durability option to disable
+ // all syncing.
+
+ logNotSynced =
+ PropertyUtil.getSystemBoolean(
+ Property.STORAGE_LOG_NOT_SYNCED);
+
+ if (logNotSynced)
+ {
+ isWriteSynced = false;
+ Monitor.logMessage("Performance.logNotSynced = true");
+ }
+ }
// try to access the log
// if it doesn't exist, create it.
Modified: db/derby/code/trunk/java/engine/org/apache/derby/loc/messages_en.properties
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/loc/messages_en.properties?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/loc/messages_en.properties (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/loc/messages_en.properties Tue Feb 7 04:07:14 2006
@@ -228,7 +228,7 @@
XSLA7.D=Cannot redo operation {0} in the log.
XSLA6.D=Cannot recover the database.
XSLA5.D=Cannot read log stream for some reason to rollback transaction {0}.
-XSLA4.D=Cannot write to the log, most likely the log is full. Please delete unnecessary files. It is also possible that the file system is read only, or the disk has failed, or some other problems with the media.
+XSLA4.D=Cannot write to the log, most likely the log is full. Please delete unnecessary files. It is also possible that the file system is read only, or the disk has failed, or some other problems with the media.
XSLA3.D=Log Corrupted, has invalid data in the log stream.
XSLA2.D=System will shutdown, got I/O Exception while accessing log file.
XSLA1.D=Log Record has been sent to the stream, but it cannot be applied to the store (Object {0}). This may cause recovery problems also.
@@ -338,6 +338,7 @@
XSDFD.S=Different page image read on 2 I/Os on Page {0}, first image has incorrect checksum, second image has correct checksum. Page images follows: {1} {2}
XSDFF.S=The requested operation failed due to an unexpected exception.
XSDFH.S=Cannot backup the database, got an I/O Exception while writing to the backup container file {0}.
+XSDFI.S=Error encountered while trying to write data to disk during database recovery. Check that the database disk is not full. If it is then delete unnecessary files, and retry connecting to the database. It is also possible that the file system is read only, or the disk has failed, or some other problem with the media. System encountered error while process page {0}.
# java/com/ibm/db2j/impl/Database/Storage/RawStore/Data/FileSystem
# database errors.
Modified: db/derby/code/trunk/java/shared/org/apache/derby/shared/common/reference/SQLState.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/shared/org/apache/derby/shared/common/reference/SQLState.java?rev=375583&r1=375582&r2=375583&view=diff
==============================================================================
--- db/derby/code/trunk/java/shared/org/apache/derby/shared/common/reference/SQLState.java (original)
+++ db/derby/code/trunk/java/shared/org/apache/derby/shared/common/reference/SQLState.java Tue Feb 7 04:07:14 2006
@@ -503,6 +503,8 @@
String FILE_UNEXPECTED_EXCEPTION = "XSDFF.S";
String FILE_ILLEGAL_ENCRYPTED_PAGE_SIZE = "XSDFG.S";
String BACKUP_FILE_IO_ERROR = "XSDFH.S";
+ String FILE_NEW_PAGE_DURING_RECOVERY = "XSDFI.S";
+
/*
** RawStore - Data.FSLDemo transaction exceptions
*/
Re: svn commit: r375583 - in /db/derby/code/trunk/java:
engine/org/apache/derby/iapi/reference/
engine/org/apache/derby/iapi/services/cache/
engine/org/apache/derby/impl/services/cache/
engine/org/apache/derby/impl/store/raw/data/ engine/org/apache/derby/i...
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, Mike. For my own edification more than anything I scanned this commit.
I'm curious, why do the property names start with "db2j.storage" instead
of "derby.storage"? Is that a historical thing? Maybe we should
change this, since these are user-visible properties?