You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Suresh Thalamati (JIRA)" <de...@db.apache.org> on 2005/05/18 21:32:55 UTC
[jira] Created: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
---------------------------------------------------------------------------------------------------------
Key: DERBY-298
URL: http://issues.apache.org/jira/browse/DERBY-298
Project: Derby
Type: Bug
Components: Store
Versions: 10.0.2.1
Reporter: Suresh Thalamati
Priority: Minor
If the system crashes after a rollforward backup; last log file
is empty(say log2.dat). On next crash-recovery system ignores the empty log
file and starts writing to the previous log(say log1.dat),
even thought there was successfule log file switch before the crash.
The reason I belive it is done this way to avoid special
handling of crashes during the log switch process.
Problem is on rollfroward restore from a backup log1.dat will get overwritten
from the copy in the backup, so any transaction that got added to log1.dat
after the backup was taken will be lost.
One possible solution that comes to my mind to solve this problem is
1) check if an empty a log file exist after a redo crash-recovery , if
the log archive mode is enabled.
2) If it exists , delete and do log file switch again
Repro:
connect 'jdbc:derby:wombat;create=true';
create table t1(a int ) ;
insert into t1 values(1) ;
insert into t1 values(2) ;
call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
'extinout/mybackup', 0);
--crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
connect 'jdbc:derby:wombat';
insert into t1 select a*2 from t1 ;
insert into t1 select a*2 from t1 ;
insert into t1 select a*2 from t1 ;
insert into t1 select a*2 from t1 ;
insert into t1 select a*2 from t1 ;
insert into t1 select a*2 from t1 ;
insert into t1 select a*2 from t1 ;
select count(*) from t1 ;
--exit from jvm and restore from backup
connect
'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
Øystein Grøvlen updated DERBY-298:
----------------------------------
Attachment: derby-298a.diff
This new patch should address the issues from the last review. The only file that has changed from the earlier patch is java/engine/org/apache/derby/impl/store/raw/log/Scan.java, which now contains the following changes:
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff, derby-298a.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [PATCH] (DERBY-298) rollforward will not work correctly if the
system happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ØG" == Øystein Grøvlen <Oy...@Sun.COM> writes:
>>>>> "ØG(" == Øystein Grøvlen (JIRA) <de...@db.apache.org> writes:
ØG> ØG(> [ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
ØG> ØG(> Ãystein Grøvlen updated DERBY-298:
ØG> ØG(> ----------------------------------
ØG> ØG(> Attachment: derby-298.diff
ØG> ØG(> The attached patch fixes the bug by setting the logEnd after recovery to the beginning of the new empty log file instead of the end of the previous file.
ØG> I guess Mike and/or Suresh should take a look at this patch.
ØG> (I will be on vacation for the next one and a half weeks. So you have
ØG> some time to respond.)
Forgot to mention that I have run derbyall with no errors.
--
Øystein
Re: [PATCH] (DERBY-298) rollforward will not work correctly if the
system happens to crash immediately after rollforward backup.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I will look at committing this change. I would appreciate it if Suresh
also reviews and let's us know what he thinks.
Øystein Grøvlen wrote:
>>>>>>"ØG(" == Øystein Grøvlen (JIRA) <de...@db.apache.org> writes:
>
>
> ØG(> [ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
> ØG(> Øystein Grøvlen updated DERBY-298:
> ØG(> ----------------------------------
>
> ØG(> Attachment: derby-298.diff
>
> ØG(> The attached patch fixes the bug by setting the logEnd after recovery to the beginning of the new empty log file instead of the end of the previous file.
> I guess Mike and/or Suresh should take a look at this patch.
>
> (I will be on vacation for the next one and a half weeks. So you have
> some time to respond.)
>
[PATCH] (DERBY-298) rollforward will not work correctly if the system
happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ØG(" == Øystein Grøvlen (JIRA) <de...@db.apache.org> writes:
ØG(> [ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
ØG(> Ãystein Grøvlen updated DERBY-298:
ØG(> ----------------------------------
ØG(> Attachment: derby-298.diff
ØG(> The attached patch fixes the bug by setting the logEnd after recovery to the beginning of the new empty log file instead of the end of the previous file.
I guess Mike and/or Suresh should take a look at this patch.
(I will be on vacation for the next one and a half weeks. So you have
some time to respond.)
--
Øystein
[jira] Updated: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
Øystein Grøvlen updated DERBY-298:
----------------------------------
Attachment: derby-298.diff
The attached patch fixes the bug by setting the logEnd after recovery to the beginning of the new empty log file instead of the end of the previous file.
The patch contains changes to the following files:
M java/engine/org/apache/derby/impl/store/raw/log/FileLogger.java
- At the end of the redo scan, if the scan stopped in a file succeeding the file of the last log record, update logEnd to this position.
- Change assert to allow logEnd to be in a newer file than that of the last log record.
M java/engine/org/apache/derby/impl/store/raw/log/Scan.java
- Introduced new variable newFileStart which will only have a valid LogInstant value when the scan is at the header of the file.
- When a new file is entered, set newFileStart to the first possible LogInstant of this file (end of header).
- When a log record is encountered, set newFileStart to INVALID_LOG_INSTANT.
- Changed getLogRecordEnd() to return newFileStart if that is valid (i.e., scan is at the start of a file)
- Removed comment about not starting to write to the new empty log file, since that is not true anymore.
A java/testing/org/apache/derbyTesting/functionTests/tests/store/RecoveryAfterBackup_app.properties
- Test properties
M java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant
- Added new property files
A java/testing/org/apache/derbyTesting/functionTests/tests/store/RecoveryAfterBackupSetup_app.properties
- Test properties.
- useextdirs=true needed so the backup is placed somewhere the next test can find it.
A java/testing/org/apache/derbyTesting/functionTests/tests/store/RecoveryAfterBackup.java
- Test that is supposed to be run after RecoveryAfterBackupSetup.java.
- Does recovery, updates the database, shutdowns the database, and does roll-forward restore.
- Checks that updates made after recovery is reflected in the database after roll-forward restore.
A java/testing/org/apache/derbyTesting/functionTests/tests/store/RecoveryAfterBackupSetup.java
- Test that does the preparation for the RecoveryAfterBackup test.
- Inserts a few records, makes a backup, and stops without shutting down.
M java/testing/org/apache/derbyTesting/functionTests/harness/RunTest.java
- For tests where the database is not deleted at the end of the test, do not delete the external directories either.
- This is necessary to be able to access the backup in suceeding tests.
A java/testing/org/apache/derbyTesting/functionTests/master/RecoveryAfterBackupSetup.out
- Test output
A java/testing/org/apache/derbyTesting/functionTests/master/RecoveryAfterBackup.out
- Test output
MM java/testing/org/apache/derbyTesting/functionTests/suites/storerecovery.runall
- Added tests to storerecovery suite.
- Changed property eol-style.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
Øystein Grøvlen updated DERBY-298:
----------------------------------
Attachment: derby-298b.diff
derby-298b.diff is an updated patch that fixes the issues raised by Suresh in his latest review.
The storeall testsuite has been run without errors. Are currently running the derbyall testsuite.
Will notify list later today if derbyall should fail.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff, derby-298a.diff, derby-298b.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12361817 ]
Øystein Grøvlen commented on DERBY-298:
---------------------------------------
Suresh,
You are right that with my latest version of the patch,
getLogRecorEnd() will only return INVALID_LOG_INSTANT when the scan
did not get out of the starting blocks. Regardsless of that, the
second part of the if test will always be false when end is
INVALID_LOG_INSTANT.
I will remove the unecessary check and fix the copyright notices.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff, derby-298a.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12362691 ]
Daniel John Debrunner commented on DERBY-298:
---------------------------------------------
This patch added two new tests, RecoveryAfterBackup and RecoveryAfterBackupSetup, both of which are being run without a SecurityManager due to the noSecurityManager=true in their _app.properties files.
Why are these tests excluded from running with a SecurityManager?
It should be rare that a test is to not to be run with a SecurityManager, if such a need arises the noSecurityManager=true must be commented to indicate why this is the case.
The default for any new tests in the existing harness should be to run with the SecurityManager.
There are a handful of tests that have noSecurityManager=true with no comment or a comment that says needs investigating, I'm working on all of these since they are exisiting tests, but it's not my itch to clean up new tests.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Fix For: 10.1.3.0, 10.2.0.0
> Attachments: derby-298.diff, derby-298a.diff, derby-298b.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12313185 ]
Øystein Grøvlen commented on DERBY-298:
---------------------------------------
This is further discussed in the following thread: http://thread.gmane.org/gmane.comp.apache.db.derby.devel/4170
My current view is that as long as the header it valid, it should be safe to use a new empty log file after recovery. Unless someone protests, I will change the recovery code to accept the newest log file even if its empty.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen wrote:
>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>>>>>>
>>>>>>
>
> ST> derby has two types of log files , one that works in RWS mode with a
> ST> preallocated log file ,
>
>What is RWS mode?
>
>
RWS mode is writing transction log using the write sync mechanism
supported by java.io.RandomaccessFile from jdk1.4.2 onwards, when a
file is opened with "rws" , all writes to the file are synced to the
disk on the write call itself.
> ST> and one which uses file sync with out preallocation. In case of
> ST> preallocation , zeros are written to the log file to the length
> ST> specified by the logSwitchInterval (Default is 1 MB) , it is also
> ST> configurable by the user.
>
>What determines which mode is used?
>
>
Write sync is the default mode when derby is run on jvm after
jdk1.4.2. users can also specify the engine to use
file sync mode , for example to avoid Derby-1 problem on Mac-OS.
> ST> This problem can also be fixed by writing dummy log record.
> ST> Filelogger.redo() code has to be fixed to understand this, currently
> ST> it looks at the logEnd only when a good log records is read.
>
>I was thinking that the "dummy" log record should be a "good" log
>record so that the current implementation of redo() need not change.
> ST> Another possible solution I was thinking to identify whether a log
> ST> switch is good or not is by writing a INT (4 bytes of zeros ) after
> ST> log file initialization. As 512 bytes writes suppose to be atomic.
> ST> if (log file length > = LOG_FILE_HEADER_SIZE(24) +4) then the log
> ST> switch before the crash can be fixed as good one and fix the scan code
> ST> to use the empty log file instead of writing to the previous log file.
>
>I am not sure I understood this. Do you suggest writing the INT just
>after the header?
>
Yes. If the header is written to the disk completely, then, it is
safe to say log switch is completed successfully.
Basically:
1) Write the header and do a file sync as it done currently.
2) Wrie a INT(4 bytes of zeros) and do a file sync.
On scan if the log file size is >= LOG_FILE_HEADER_SIZE(24) +4 ,
then the log file switch was successful before the crash.
It can be used as the next log file.
> Would this work for preallocated log files?
>
I think so, preallocation is done only for performance reasons. For some reason if the preallocation did not complete, recovery should work fine. just that log writes will be slower until the next switch.
>Would
>not the length always be 1 MB (default) in that case?
>
>
>
Not sure I understand this question completely. Length will not be 1MB
(default) If the crash occurs after log file initialization , but before
preallocation. It will be just LOG_FILE_HEADER_SIZE(24) +4 .
Thanks
-suresh
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> derby has two types of log files , one that works in RWS mode with a
ST> preallocated log file ,
What is RWS mode?
ST> and one which uses file sync with out preallocation. In case of
ST> preallocation , zeros are written to the log file to the length
ST> specified by the logSwitchInterval (Default is 1 MB) , it is also
ST> configurable by the user.
What determines which mode is used?
ST> This problem can also be fixed by writing dummy log record.
ST> Filelogger.redo() code has to be fixed to understand this, currently
ST> it looks at the logEnd only when a good log records is read.
I was thinking that the "dummy" log record should be a "good" log
record so that the current implementation of redo() need not change.
ST> Another possible solution I was thinking to identify whether a log
ST> switch is good or not is by writing a INT (4 bytes of zeros ) after
ST> log file initialization. As 512 bytes writes suppose to be atomic.
ST> if (log file length > = LOG_FILE_HEADER_SIZE(24) +4) then the log
ST> switch before the crash can be fixed as good one and fix the scan code
ST> to use the empty log file instead of writing to the previous log file.
I am not sure I understood this. Do you suggest writing the INT just
after the header? Would this work for preallocated log files? Would
not the length always be 1 MB (default) in that case?
--
Øystein
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_66168 ]
>
>Øystein Grøvlen commented on DERBY-298:
>---------------------------------------
>
>Looking at the code, I became a bit confused about the definition of an empty log file. Scan.getNextRecordForward contains debug output when it detects an empty log file. It will then return without setting knownGoodLogEnd. Hence, new log records will be written to the end of the previous file. As Suresh says this is probably to be able to handle crashes during log switch.
>
>However, this is not what happens when I run the recovery part of the example in this report. Since, currentLogFileLength is a large number, it detects "zapped log end on log file", goes on to the next file, which does not exist, and returns. (Who sets the length of a log file? Is this maximum size until a log switch is performed?) The effect is the same, but this can not be used to detect an empty log file and apply the solution proposed by Suresh. Instead, one would have to do some hairy file handling at a later stage.
>
>
>
derby has two types of log files , one that works in RWS mode with a
preallocated log file ,
and one which uses file sync with out preallocation. In case of
preallocation , zeros are written to the log file to the length
specified by the logSwitchInterval (Default is 1 MB) , it is also
configurable by the user.
Empty log file can not be identified based on the length alone. . Only
way to declare a log file is empty/fresh is when No log records found
in the file during recover scan. As u noticed because it is a
preallocated file , when scan finds zeros on first after zeros, it
declares that there are no more log records. Any fix for this problem
has to handle both preallocated and non preallocated case .
Actually, I don't like idea of creating a new log file on boot even
for special conditions,
because spending more time than required does not make out users happy
! If you can find a fix that does not require a new log creation, it
will be great.
>An alternative way to fix this would be to just create a dummy log record in the new log file as part of the backup command. This would make the redo scan end in the new log file. However, this will not work for those who do backup with OS-commands (i.e., copy the files directly).
>
>
Backup with OS command should not be a problem , because there is no
support to perform roll forward recovery with these type of backups. If
it is a just a plain backup , it does not matter how logs are
written after the backup, because they are never used to do a restore.
This problem can also be fixed by writing dummy log record.
Filelogger.redo() code has to be fixed to understand this, currently it
looks at the logEnd only when a good log records is read.
>I would also think it should be possible to do the log switch in such a way that it is possible to detect during recovery whether the log switch had completed or not. If this was the case, one could just set knownGoodLogEnd of the redo scan to the start of the empty file if the log switch was completed. Does anyone know if this is possible?
>
>
>
Yes. It should be, with some changes to do redo/scan code.
Another possible solution I was thinking to identify whether a log
switch is good or not is by writing a INT (4 bytes of zeros ) after
log file initialization. As 512 bytes writes suppose to be atomic. if
(log file length > = LOG_FILE_HEADER_SIZE(24) +4) then the log switch
before the crash can be fixed as good one and fix the scan code to
use the empty log file instead of writing to the previous log file.
Thanks
-suresht
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_66168 ]
Øystein Grøvlen commented on DERBY-298:
---------------------------------------
Looking at the code, I became a bit confused about the definition of an empty log file. Scan.getNextRecordForward contains debug output when it detects an empty log file. It will then return without setting knownGoodLogEnd. Hence, new log records will be written to the end of the previous file. As Suresh says this is probably to be able to handle crashes during log switch.
However, this is not what happens when I run the recovery part of the example in this report. Since, currentLogFileLength is a large number, it detects "zapped log end on log file", goes on to the next file, which does not exist, and returns. (Who sets the length of a log file? Is this maximum size until a log switch is performed?) The effect is the same, but this can not be used to detect an empty log file and apply the solution proposed by Suresh. Instead, one would have to do some hairy file handling at a later stage.
An alternative way to fix this would be to just create a dummy log record in the new log file as part of the backup command. This would make the redo scan end in the new log file. However, this will not work for those who do backup with OS-commands (i.e., copy the files directly).
I would also think it should be possible to do the log switch in such a way that it is possible to detect during recovery whether the log switch had completed or not. If this was the case, one could just set knownGoodLogEnd of the redo scan to the start of the empty file if the log switch was completed. Does anyone know if this is possible?
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
Øystein Grøvlen updated DERBY-298:
----------------------------------
Comment: was deleted
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff, derby-298a.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
Øystein Grøvlen resolved DERBY-298:
-----------------------------------
Fix Version: 10.2.0.0
10.1.3.0
Resolution: Fixed
Verified that the added test run without failure.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Fix For: 10.1.3.0, 10.2.0.0
> Attachments: derby-298.diff, derby-298a.diff, derby-298b.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> If possible I would like to see a solution that does not
MM> depend on a special log record being first in the new log. At
MM> the level of specific log files, it does not know much about
MM> log records. It would be nice if anything that dealt with log
MM> records just saw them as a single sequence of log records
MM> which just happen to be split across multiple files in this
MM> implementation. Putting stuff in the per log file hdr is fine
MM> as this is specific to this implementation.
I understand and agree. We should separate the implementation of log
files from the implementation of a stream of log records.
MM> Couldn't step 11 be another bit in the file hdr?
Yes, I think it could. However, I am starting to think that special
handling of crashed during log switching is not really necessary.
Suresh wrote in the bug report:
> On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
Looking at the steps of log switching again:
>> 1. Check if file with next file number exists. If so, delete it
>> 2. Open new log file
>> 3. Initialize file (write header) and sync
>> 4. Write end marker to old file
>> 5. Flush old file to disk
>> 6. Close old file
>> 7. Preallocate space in new log file and sync
>> 8. Close new file
>> 9. Reopen new file in RWS mode
>> 10. Go to current end position (after header).
Why are we concered about crashes after step 3? According to Suresh
step 7 (preallocation) is not strictly necessary. Hence, the usage of
the new log file is not dependent on steps 4 - 10. The recovery
process will have to deal with an incomplete old file anyway. What is
the problem with using the new file after recovery as long as the
header is valid?
--
Øystein
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Suresh Thalamati <su...@gmail.com>.
Mike Matrigali wrote:Any requirement to put a specific log record "next"
in the stream is
<snip>
>I believe currently log recordsare being written to the in memory log buffer while the switch is
>happening (suresh, is this right?).
>
Not to the log buffers (in LogAccessFile.java) that are common to whole
database for sure. Before writing into the log buffers. an LS is
required. LS in Derby is based on the log file that is going to be
used to write the log record, so until the log switch is complete,
all writes to the log buffers will be blocked.
....
<snip>
-surest
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
If possible I would like to see a solution that does not depend on a
special log record being first in the new log. At the level of specific
log files, it does not know much about log records. It would be nice if
anything that dealt with log records just saw them as a single sequence
of log records which just happen to be split across multiple files in
this implementation. Putting stuff in the per log file hdr is fine as
this is specific to this implementation.
Any requirement to put a specific log record "next" in the stream is
likely to cause us to stall all the threads in the system. Something
missing from your steps is what is going on concurrently in the rest
of the system while this is happening. I believe currently log records
are being written to the in memory log buffer while the switch is
happening (suresh, is this right?). Once it goes into the log buffer
it has an address that will make it hard to put a different record
"before" it.
Couldn't step 11 be another bit in the file hdr?
Øystein Grøvlen wrote:
>>>>>>"MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
>
>
> MM> As you suggest I think there needs to be some way for recovery to
> MM> determine a successful log switch has happened and to not log to the
> MM> old file. I don't think a log record for the switch works as you need
> MM> to read the log to apply the record, but I think this is problematical
> MM> as you are trying to determine validity of the log by looging at record
> MM> in the log. I think some approach as you and suresh suggest with
> MM> markers in the header of the individual log files is a better
> MM> approach.
>
> Assuming sync write, the steps to do a log switch is:
>
> 1. Check if file with next file number exists. If so, delete it
> 2. Open new log file
> 3. Initialize file (write header) and sync
> 4. Write end marker to old file
> 5. Flush old file to disk
> 6. Close old file
> 7. Preallocate space in new log file and sync
> 8. Close new file
> 9. Reopen new file in RWS mode
> 10. Go to current end position (after header).
>
> Basically if a crash occurs before step 3 is finished, the log file
> will just be ignored on recovery because the validation of the header
> fails. Should it fail after step 3, redo processing will try to
> access the first log rec of the file. If there is none, the last good
> log record pointer will not be updated and logging will occur in the
> previous file. This is OK if the crash occurred before the log switch
> was completed since then the operation that initiated the log switch
> would not complete either (e.g., a backup would fail). As we have
> agreed on, we need something to tell recovery that the log switch was
> completed.
>
> However, there is currently nothing that prevents recovery from
> interpreting garbage succeeding the header as a log record and
> wrongfully conclude that the log switch was completed. Even if it is
> not very likely that this would happen, I suggest that to be safe we
> write an integer 0 after the header as part of step 3. In a new step
> 11, we could then write a log switch log record which length would
> overwrite the 0 written in step 3. This way, depending on during
> which step the crash occurred, the following would happen during the
> redo scan using the existing recovery implementation:
>
> Crash during step 1 to 3: New file is not valid, continue to use old log file
> Crash during step 4 to 11: New file is empty, continue to use old log file
> Crash after step 11: Process log switch log record, next log record
> will be allocated after this record.
>
> I think this is the behavior we want.
>
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> As you suggest I think there needs to be some way for recovery to
MM> determine a successful log switch has happened and to not log to the
MM> old file. I don't think a log record for the switch works as you need
MM> to read the log to apply the record, but I think this is problematical
MM> as you are trying to determine validity of the log by looging at record
MM> in the log. I think some approach as you and suresh suggest with
MM> markers in the header of the individual log files is a better
MM> approach.
Assuming sync write, the steps to do a log switch is:
1. Check if file with next file number exists. If so, delete it
2. Open new log file
3. Initialize file (write header) and sync
4. Write end marker to old file
5. Flush old file to disk
6. Close old file
7. Preallocate space in new log file and sync
8. Close new file
9. Reopen new file in RWS mode
10. Go to current end position (after header).
Basically if a crash occurs before step 3 is finished, the log file
will just be ignored on recovery because the validation of the header
fails. Should it fail after step 3, redo processing will try to
access the first log rec of the file. If there is none, the last good
log record pointer will not be updated and logging will occur in the
previous file. This is OK if the crash occurred before the log switch
was completed since then the operation that initiated the log switch
would not complete either (e.g., a backup would fail). As we have
agreed on, we need something to tell recovery that the log switch was
completed.
However, there is currently nothing that prevents recovery from
interpreting garbage succeeding the header as a log record and
wrongfully conclude that the log switch was completed. Even if it is
not very likely that this would happen, I suggest that to be safe we
write an integer 0 after the header as part of step 3. In a new step
11, we could then write a log switch log record which length would
overwrite the 0 written in step 3. This way, depending on during
which step the crash occurred, the following would happen during the
redo scan using the existing recovery implementation:
Crash during step 1 to 3: New file is not valid, continue to use old log file
Crash during step 4 to 11: New file is empty, continue to use old log file
Crash after step 11: Process log switch log record, next log record
will be allocated after this record.
I think this is the behavior we want.
--
�ystein
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
comments below
Øystein Grøvlen wrote:
>>>>>>"MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
>
>
> MM> I spent some time thinking about this and I think the dummy record
> MM> approach works in the current system assuming the code is as follows:
>
> MM> o request for roll forward backup starts
> MM> o all subsequent write requests suspended
> MM> o log switch initiated by backup request
> MM> o dummy record written
> MM> o backup takes place, any failure during backup somehow marks backup as
> MM> failed.
> MM> o backup succeeds, rest of system is allowed to continue writing.
>
> Except from the dummy record part, this seems to be the current
> sequence. (Maybe we should call it "start of backup log record"
> instead of dummy record). I am bit uncertain about whether a failure
> marks backup as failed. I could only find code for the case where an
> old backup existed at the same location. In that case the old backup
> will be restored if the current backup fails. There does not seem to
> be any error handling for the case where no previous backup exists.
>
> MM> Will a similar approach work when we support real online backup, where
> MM> threads are allowed to continue writing to the log files during the
> MM> backup?
>
> I am not sure how Suresh plans to implement restore of a real online
> backup. Today, the purpose of a log switch is to separate the log
> records that need to be copied during backup from later log records.
> When online backup is performed, one will also need log generated
> during the backup. Maybe it then would make sense to do the log
> switch at the end of the backup. Anyhow, there will still be a need
> for a dummy record in the new log file since there is no guarantee
> that there will be any concurrent updates.
>
> MM> Also at some point we should be allowing copies of each log file as it
> MM> is finished - or at least as some unit of work is finished - maybe log
> MM> file, maybe checkpoint. In that case would we need to write a dummy
> MM> record in every file?
>
> Are you thinking about some kind of archival service where log files
> are automatically copied when the database switches to a new log file?
> I guess what is important is to make sure that after recovery, the
> database does not start to log to a file that has already been copied.
> This makes me think that this is a general recovery problem, and
> should not just be fixed in the context of backup. We should then fix
> recovery to redo any log switch that was completed before the crash.
> One way to do this is to write a log record to the new log file on
> every log switch. This would work with the current recovery
> implementation. An alternative is to write some bytes somewhere in
> the new log file in order to signal that the log switch is completed
> and change the redo scan to handle that.
>
Yes, at some point I think we should implement some sort of archival
service as described. Probably allowing users to override the copy
if they want so they could handle copying the file to another machine,
a different device, ....
And I also believe this may be a general recovery problem and should
be solved not just in context of current backup. It may be the case
that current backup is only place that sees the problem, but would
like to get agreement on best solution going forward.
As you suggest I think there needs to be some way for recovery to
determine a successful log switch has happened and to not log to the
old file. I don't think a log record for the switch works as you need
to read the log to apply the record, but I think this is problematical
as you are trying to determine validity of the log by looging at record
in the log. I think some approach as you and suresh suggest with
markers in the header of the individual log files is a better approach.
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> I spent some time thinking about this and I think the dummy record
MM> approach works in the current system assuming the code is as follows:
MM> o request for roll forward backup starts
MM> o all subsequent write requests suspended
MM> o log switch initiated by backup request
MM> o dummy record written
MM> o backup takes place, any failure during backup somehow marks backup as
MM> failed.
MM> o backup succeeds, rest of system is allowed to continue writing.
Except from the dummy record part, this seems to be the current
sequence. (Maybe we should call it "start of backup log record"
instead of dummy record). I am bit uncertain about whether a failure
marks backup as failed. I could only find code for the case where an
old backup existed at the same location. In that case the old backup
will be restored if the current backup fails. There does not seem to
be any error handling for the case where no previous backup exists.
MM> Will a similar approach work when we support real online backup, where
MM> threads are allowed to continue writing to the log files during the
MM> backup?
I am not sure how Suresh plans to implement restore of a real online
backup. Today, the purpose of a log switch is to separate the log
records that need to be copied during backup from later log records.
When online backup is performed, one will also need log generated
during the backup. Maybe it then would make sense to do the log
switch at the end of the backup. Anyhow, there will still be a need
for a dummy record in the new log file since there is no guarantee
that there will be any concurrent updates.
MM> Also at some point we should be allowing copies of each log file as it
MM> is finished - or at least as some unit of work is finished - maybe log
MM> file, maybe checkpoint. In that case would we need to write a dummy
MM> record in every file?
Are you thinking about some kind of archival service where log files
are automatically copied when the database switches to a new log file?
I guess what is important is to make sure that after recovery, the
database does not start to log to a file that has already been copied.
This makes me think that this is a general recovery problem, and
should not just be fixed in the context of backup. We should then fix
recovery to redo any log switch that was completed before the crash.
One way to do this is to write a log record to the new log file on
every log switch. This would work with the current recovery
implementation. An alternative is to write some bytes somewhere in
the new log file in order to signal that the log switch is completed
and change the redo scan to handle that.
--
Øystein
Re: [jira] Commented: (DERBY-298) rollforward will not work correctly
if the system happens to crash immediately after rollforward backup.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I spent some time thinking about this and I think the dummy record
approach works in the current system assuming the code is as follows:
o request for roll forward backup starts
o all subsequent write requests suspended
o log switch initiated by backup request
o dummy record written
o backup takes place, any failure during backup somehow marks backup as
failed.
o backup succeeds, rest of system is allowed to continue writing.
Will a similar approach work when we support real online backup, where
threads are allowed to continue writing to the log files during the
backup?
Also at some point we should be allowing copies of each log file as it
is finished - or at least as some unit of work is finished - maybe log
file, maybe checkpoint. In that case would we need to write a dummy
record in every file?
Øystein Grøvlen (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_66209 ]
>
> Øystein Grøvlen commented on DERBY-298:
> ---------------------------------------
>
> On second thoughts, recovery after an OS-command backup is not an issue since the database will be shut down when the backup is performed. So, why not insert a "dummy" record in the new log file when doing backup? Then, no changes would be needed to the recovery mechanism.
>
>
>
>>rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
>>---------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-298
>> URL: http://issues.apache.org/jira/browse/DERBY-298
>> Project: Derby
>> Type: Bug
>> Components: Store
>> Versions: 10.0.2.1
>> Reporter: Suresh Thalamati
>> Assignee: Øystein Grøvlen
>> Priority: Minor
>
>
>>If the system crashes after a rollforward backup; last log file
>>is empty(say log2.dat). On next crash-recovery system ignores the empty log
>>file and starts writing to the previous log(say log1.dat),
>>even thought there was successfule log file switch before the crash.
>>The reason I belive it is done this way to avoid special
>>handling of crashes during the log switch process.
>>Problem is on rollfroward restore from a backup log1.dat will get overwritten
>>from the copy in the backup, so any transaction that got added to log1.dat
>>after the backup was taken will be lost.
>>
>>One possible solution that comes to my mind to solve this problem is
>> 1) check if an empty a log file exist after a redo crash-recovery , if
>> the log archive mode is enabled.
>> 2) If it exists , delete and do log file switch again
>>
>>Repro:
>>connect 'jdbc:derby:wombat;create=true';
>>create table t1(a int ) ;
>>insert into t1 values(1) ;
>>insert into t1 values(2) ;
>>call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
>> 'extinout/mybackup', 0);
>>--crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
>>connect 'jdbc:derby:wombat';
>>insert into t1 select a*2 from t1 ;
>>insert into t1 select a*2 from t1 ;
>>insert into t1 select a*2 from t1 ;
>>insert into t1 select a*2 from t1 ;
>>insert into t1 select a*2 from t1 ;
>>insert into t1 select a*2 from t1 ;
>>insert into t1 select a*2 from t1 ;
>>select count(*) from t1 ;
>>--exit from jvm and restore from backup
>>connect
>>'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
>>select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
>
>
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_66209 ]
Øystein Grøvlen commented on DERBY-298:
---------------------------------------
On second thoughts, recovery after an OS-command backup is not an issue since the database will be shut down when the backup is performed. So, why not insert a "dummy" record in the new log file when doing backup? Then, no changes would be needed to the recovery mechanism.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12315265 ]
Suresh Thalamati commented on DERBY-298:
----------------------------------------
Hi Øystein,
I could not understand from the changes, If the new log file will be recognized in the following two cases:
1) After the log switch, if the first log record to the file gets partially written on a preallocated log file
2) After the log swith , if the first log write is written out of order incompletely and the checksum check verification fails.
In Scan.java : newFileStart gets set to LogCounter.INVALID_LOG_INSTANT, immediately
after reading the instant , but the partial record verification and checksum checks happen
after that and valid logEnd value still refers to previous log file .
and also it might be good idea to make sure all the fields written in the initialization of the log file are correct before using the new log file during recovery: initialization writes 4 fileds seperately ,
whereas verification only looks at 3 fields.
LogToFIle.java() : initLogFile() :
newlog.writeInt(fid);
newlog.writeInt(OBSOLETE_LOG_VERSION_NUMBER); // for silly backwards compatibility reason
newlog.writeLong(number);
newlog.writeLong(prevLogRecordEndInstant);
whereas LogTOFIle: private boolean verifyLogFormat(StorageRandomAccessFile log, longnumber)
which is called before the swicth does not read/verify "prevLogRecordEndInstant" .
Thanks
-suresh
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12361808 ]
Suresh Thalamati commented on DERBY-298:
----------------------------------------
Øystein ,
I reviewed the latest patch, it looks good. But I could not understand why you would need the following check ?
FileLogge.java :
+ long end = redoScan.getLogRecordEnd();
+ if (end != LogCounter.INVALID_LOG_INSTANT
If end is LogCounter.INVALID_LOG_INSTANT then logEnd is also likely to be LogCounter.INVALID_LOG_INSTANT right ?
+ && (LogCounter.getLogFileNumber(logEnd)
+ < LogCounter.getLogFileNumber(end))) {
+ logEnd = end;
+ }
In what secnario condition will be false ? If end is LogCounter.INVALID_LOG_INSTANT then logEnd is also likely to be
Another minor thing I notices is test files copyrigth notices have wrong file names :
RecoveryAfterBackup.java:
Derby - Class org.apache.derbyTesting.functionTests.store.LogChecksumSetup
RecoveryAfterBackupSetup.java:
Derby - Class org.apache.derbyTesting.functionTests.store.LogChecksumSetup
Thanks
-suresh
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff, derby-298a.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12315919 ]
Øystein Grøvlen commented on DERBY-298:
---------------------------------------
Suresh,
The intent is that the new log file should be recognized after a log switch, regardless of what log records may have been written after this log switch. I see your point that if the log file contains a single corrupt record, the log switch will not have an effect since newFileStart has been reset and logEnd has not been advanced to the new file. Thanks for detecting this.
My suggestion for fixing this, is to drop the introduction of newFileStart and just update logEnd to the start of the new log file when the log file is detected. In addition, logEnd must not be reset to INVALID_LOG_INSTANT when a scan is closed. This way it is possible to get the logEnd after the scan is closed and check whether it has been advanced since the last log record was processed.
For the verification of prevLogRecordEndInstant, I guess that the best is to verify this for all files processed during recovery. I assume that the prevLogRecordEndInstant of a file should be equal to the current logEnd when switching to this file.
Thanks for the advice, I will create a new patch.
Øystein
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [PATCH] Re: [jira] Commented: (DERBY-298) rollforward will not
work correctly if the system happens to crash immediately after rollforward
backup.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have committed the most recent patch for this issue as svn 367352.
Øystein Grøvlen wrote:
>>>>>>"MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
>
>
> MM> I just got back. I'll look at committing this, but would like to
> MM> wait for Suresh to review.
>
> The new version of the patch, derby-298b.diff, that has been uploaded
> should address issues raised by Suresh in his latest review.
>
Re: [PATCH] Re: [jira] Commented: (DERBY-298) rollforward will not
work correctly if the system happens to crash immediately after rollforward
backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> I just got back. I'll look at committing this, but would like to
MM> wait for Suresh to review.
The new version of the patch, derby-298b.diff, that has been uploaded
should address issues raised by Suresh in his latest review.
--
Øystein
Re: [PATCH] Re: [jira] Commented: (DERBY-298) rollforward will not
work correctly if the system happens to crash immediately after rollforward
backup.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I just got back. I'll look at committing this, but would like to
wait for Suresh to review.
Øystein Grøvlen wrote:
> I finally got back to this issue hat have been laying around for way
> too long. I have attached a new patch to JIRA that should address
> Suresh's review comments. Could someone please review it?
>
[PATCH] Re: [jira] Commented: (DERBY-298) rollforward will not work
correctly if the system happens to crash immediately after rollforward backup.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
I finally got back to this issue hat have been laying around for way
too long. I have attached a new patch to JIRA that should address
Suresh's review comments. Could someone please review it?
--
Øystein
>>>>> "ØG" == Øystein Grøvlen (JIRA) <de...@db.apache.org> writes:
ØG> [ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12361100 ]
ØG> Ãystein Grøvlen commented on DERBY-298:
ØG> ---------------------------------------
ØG> The recently attached patch (derby-298a.diff) addresses Suresh's
ØG> review comments. The only major change from the previous patch is in
ØG> java/engine/org/apache/derby/impl/store/raw/log/Scan.java. The changes
ØG> to this file compared to the current head of trunk are:
ØG> - When a new log file is entered, check that the header of this
ØG> file refers to the end of the last log record of the previous log
ØG> file. If not, stop the scan.
ØG> - If the header was consistent, update knowGoodLogEnd to the first
ØG> possible LogInstant of this file (end of header).
ØG> - close() no longer reset knownGoodLogEnd since it is needed by
ØG> FileLogger after the scan is closed.
ØG> - Changed comment of getLogRecordEnd() to reflect that it can be
ØG> used after the scan is closed, and that it at that time may
ØG> return the start of an empty log file.
ØG> - Removed comment about not starting to write to the new empty log
ØG> file, since that is not true anymore.
ØG> In addition, the property files for the tests have been updated so
ØG> they are run without the security manager.
ØG> derbyall has been run with no new failures.
[jira] Commented: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=comments#action_12361100 ]
Øystein Grøvlen commented on DERBY-298:
---------------------------------------
The recently attached patch (derby-298a.diff) addresses Suresh's
review comments. The only major change from the previous patch is in
java/engine/org/apache/derby/impl/store/raw/log/Scan.java. The changes
to this file compared to the current head of trunk are:
- When a new log file is entered, check that the header of this
file refers to the end of the last log record of the previous log
file. If not, stop the scan.
- If the header was consistent, update knowGoodLogEnd to the first
possible LogInstant of this file (end of header).
- close() no longer reset knownGoodLogEnd since it is needed by
FileLogger after the scan is closed.
- Changed comment of getLogRecordEnd() to reflect that it can be
used after the scan is closed, and that it at that time may
return the start of an empty log file.
- Removed comment about not starting to write to the new empty log
file, since that is not true anymore.
In addition, the property files for the tests have been updated so
they are run without the security manager.
derbyall has been run with no new failures.
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
> Attachments: derby-298.diff, derby-298a.diff
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (DERBY-298) rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-298?page=all ]
Øystein Grøvlen reassigned DERBY-298:
-------------------------------------
Assign To: Øystein Grøvlen
> rollforward will not work correctly if the system happens to crash immediately after rollforward backup.
> ---------------------------------------------------------------------------------------------------------
>
> Key: DERBY-298
> URL: http://issues.apache.org/jira/browse/DERBY-298
> Project: Derby
> Type: Bug
> Components: Store
> Versions: 10.0.2.1
> Reporter: Suresh Thalamati
> Assignee: Øystein Grøvlen
> Priority: Minor
>
> If the system crashes after a rollforward backup; last log file
> is empty(say log2.dat). On next crash-recovery system ignores the empty log
> file and starts writing to the previous log(say log1.dat),
> even thought there was successfule log file switch before the crash.
> The reason I belive it is done this way to avoid special
> handling of crashes during the log switch process.
> Problem is on rollfroward restore from a backup log1.dat will get overwritten
> from the copy in the backup, so any transaction that got added to log1.dat
> after the backup was taken will be lost.
>
> One possible solution that comes to my mind to solve this problem is
> 1) check if an empty a log file exist after a redo crash-recovery , if
> the log archive mode is enabled.
> 2) If it exists , delete and do log file switch again
>
> Repro:
> connect 'jdbc:derby:wombat;create=true';
> create table t1(a int ) ;
> insert into t1 values(1) ;
> insert into t1 values(2) ;
> call SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(
> 'extinout/mybackup', 0);
> --crash (NO LOG RECORDS WENT IN AFTER THE BACKUP).
> connect 'jdbc:derby:wombat';
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> insert into t1 select a*2 from t1 ;
> select count(*) from t1 ;
> --exit from jvm and restore from backup
> connect
> 'jdbc:derby:wombat;rollForwardRecoveryFrom=extinout/mybackup/wombat';
> select count(*) from t1 ; -- THIS WILL GIVE INCORRECT VALUES
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira