[jira] Resolved: (DERBY-96) partial log record writes that occur because of out-of order writes need to be handled by recovery.

["Suresh Thalamati (JIRA)" <de...@db.apache.org>]

     [ http://issues.apache.org/jira/browse/DERBY-96?page=all ]
     
Suresh Thalamati resolved DERBY-96:
-----------------------------------

     Resolution: Fixed
    Fix Version: 10.1.0.0

Following changes fixed this problem:
r178494 :
small fix to make sure that log buffers are switched are correctly when the are
full, when the log checksum  feature is disabled due to a  soft upgrade.

r169737:
some  functional tests to test the transaction log checksum feature.Log corruption is simulated using a proxy storage factory that allows corruption of the log write request before being writing to the disk.  CorruptDiskStorage fact
ory by default forwards all the request to the underlying disk storage factory  except when corruption flags are  enabled.

Recovery tests need to  boot the same database many times and have to use the different Subprotocol to enable the corruption instead of the default protocol. This seems to be
possible only  by adding  a new tests suite in the current test frame work.  Add ed a new suite
called "storerecovery" , may be all future recovery tests can be added to this suite.

r164994:
changes to make softupgrade correctly with the  transaction log checksum feature in 10.1 Added  checkVersion() method  to log factory it
self,  becuase that is where the  version numbers are read  from  from the log control file , but did not export the call it to  the rawstore factory  as it is not needed now.  (This can be done easlily when there is a  need for upgrade
checks  in the other store modules..)

r159651:
This is a  patch towards implementing checksum support for transaction log to handle out of order incomplete log writes during recovery.  This patch is based on writing a checksum log record that contain checksum information for
a group of log records in the log buffers. 

Changes in this patch addresses writing checksum information to the transaction log before the log records are written and verifying the log at recovery time using the checksum information on the disk.

Writing Checksum Log Records:
Checksum log record contains checksum Algorithm, size of  the data  and the checksum value.
Added a new class to implement this log operation.

The checksum Log record is placed before the actual log data the checksum record represents. This is done by reserving the space in the log buffers and in the log file then writing into reserved buffer space the checksum log record
whenever buffer is full or it need to be written because of a flush request due to a commit. Incase of a large log records that does not fit into a single log buffer, the log records are written directly to the log file, in this case
checksum log record represents only one log record and it is written to the log file before writing the large log record directly into the log file.

In the current system the log group information is encrypted when a database is encrypted. There is no facility to identify that a log record is checksum log record without decrypting the log record. Checksum Log Record is also encrypted to work correctly with the rest of the system.

> partial log record writes that occur because of out-of order writes need to be handled by recovery.
> ---------------------------------------------------------------------------------------------------
>
>          Key: DERBY-96
>          URL: http://issues.apache.org/jira/browse/DERBY-96
>      Project: Derby
>         Type: New Feature
>   Components: Store
>     Versions: 10.0.2.1
>     Reporter: Suresh Thalamati
>     Assignee: Suresh Thalamati
>      Fix For: 10.1.0.0

>
> Incomplete log record write that occurs because of
> an out of order partial writes gets recognized as complete during
> recovery if the first sector and last sector happens to get written.
>  Current system recognizes incompletely written log records by checking
> the length of the record that is stored in the beginning and end.
>  Format the log records are written to disk is:
>   +----------+-------------+------------------+
>   | length     |  LOG RECORD |    length   |
>   +----------+-------------+------------------+
> This mechanism works fine if sectors are written in sequential manner or
> log record size is less than 2 sectors. I  believe on SCSI types disks
> order is not necessarily sequential, SCSI disk drives may sometimes do a
> reordering of the sectors to optimize the performance.  If a log record
> that spans multiple disk sectors is being written to SCISI type of
> devices,  it is possible that first and last sector written before the
> crash; If this occurs recovery system will incorrectly  interpret the
> log records was completely written and replay the record. This could
> lead to recovery errors or data corruption.
> -
> This problem also will not occur if a disk drive has write cache with a
> battery backup which will make sure I/O request will complete.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira