You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Suresh Thalamati (JIRA)" <de...@db.apache.org> on 2005/04/26 19:55:25 UTC
[jira] Created: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Need a online backup feature that does not block update operations when online backup is in progress.
--------------------------------------------------------------------------------------------------------
Key: DERBY-239
URL: http://issues.apache.org/jira/browse/DERBY-239
Project: Derby
Type: New Feature
Components: Store
Versions: 10.1.0.0
Reporter: Suresh Thalamati
Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati resolved DERBY-239:
------------------------------------
Fix Version: 10.2.0.0
Resolution: Fixed
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Fix For: 10.2.0.0
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup2.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff, onlinebackup_8.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-239) Need a online backup feature that does
not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_8.diff
This patch addresses the improvements suggested by Oystein in his review of the
first online backup patch(onlinebackup_1.diff) and also resolved some more
online backup issues.
-- fixed comments and moved a duplicate code in write page and the backup of
container into a separate method.
fixes with this patch:
-- backup of container was using the same encryption buffer as container
read/writes, this requires backup of the container and read/writes are
synchronized to avoid corrupting the encryption buffer. This patch modified
the backup of container code to use it's own temporary encryption
buffer to avoid the buffer corruption.
-- added code to prevent truncation of log during checkpoint and disabling of
log archival does not delete log files that are yet to be copied
into the backup, if backup is running in parallel.
-- In-place compress is blocked during backup and vice-versa, until the
compress nested transaction commits. This change is needed because compress
does a special checkpoint to avoid redo on the truncated pages. If backup is
running in parallel, the checkpoint backup is based on can be earlier than
checkpoint done by the truncate operation. Without the blocking during
restore from backup, recovery will fail if it needs redo ant log records on
the truncated pages.
TESTS : derbyall test suite passed on Windows XP/JDK142
It would be great if some can review and commit this patch.
svn stat:
M java\engine\org\apache\derby\impl\store\raw\log\ReadOnly.java
M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
M java\engine\org\apache\derby\impl\store\raw\RawStore.java
M java\engine\org\apache\derby\impl\store\raw\data\InputStreamContainer.jav
a
M java\engine\org\apache\derby\impl\store\raw\data\FileContainer.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseContainer.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseContainerHandle.java
M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
M java\engine\org\apache\derby\iapi\store\raw\log\LogFactory.java
M java\engine\org\apache\derby\loc\messages_en.properties
M java\shared\org\apache\derby\shared\common\reference\SQLState.java
M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest4.sql
A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest4_app.properties
A java\testing\org\apache\derbyTesting\functionTests\master\onlineBackupTest4.out
M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
Thanks
-suresh
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff, onlinebackup_8.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Thanks Øystein. I did not realize, my reply was not going derby-dev.
-suresh
Øystein Grøvlen wrote:
> Suresh,
>
> I guess you got hit by the derby-dev reply-to problem. I guess you
> would want to send your email there.
>
> I agree with what you say here. I think blocking on backup when
> loading jar files are OK. I am currently reviewing your code, and I
> have already seen that you handle the log copying correctly there.
>
> --
> Øystein
>
>
>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>
>
> ST> Øystein Grøvlen wrote:
> >>>>>>> "ST(" == Suresh Thalamati (JIRA) <de...@db.apache.org> writes:
> >>
>
> ST> ..<snip>
>
> >> ST(> Attachment: onlinebackup.html
> >> ST(> functional/design spec based on the input I got on the
> >> derby-dev list. Very good spec that seems to cover the necessary
> >> issues. I have a few
>
> >> minor comments:
>
> ST> Thanks for reviewing the spec.
>
> >> - Log files: I would think you may in some cases have to
> >> copy
>
> >> log files from before the backup checkpoint in order to be
> >> able to roll-back transactions that started before the
> >> backup. (Or am I missing something).
>
> ST> No, you are right. Log files before the backup checkpoint are needed
> ST> for rollbacks. I will edit the spec, the correct statement should be:
>
>
> ST> "All the transaction log files starting from the log file that
> ST> contains Undo Low Water Mark of the backup checkpoint to the log file
> ST> that is active at the end of the backup are copied into the backup. "
>
>
>
> >> - Jar files: What if a jar file is stored in the database
>
> >> during backup? Will this be allowed? If yes, how is it
> >> made sure that this is copied into the backup.
>
> ST> Good point. I think jar file actions have to be aware of the parallel
> ST> backup activity and vice versa, otherwise database in the backup can
> ST> possibly get into a consistent state. Because the catalog updates are
> ST> logged, but not the jar files it self, so it is possible that backup
> ST> database catalog(sys.sysfiles) will have a reference to a file that
> ST> does not exist in the backup database.
>
>
> ST> I think this problem can be solved by doing the following:
>
> ST> 1) make backup wait for all jar file activity in progress to complete,
> ST> before copying the jar files from the database to the backup.
>
> ST> 2) make any new jar file activity after the backup of jar files starts
> ST> to wait/fail.
>
>
> ST> I believe jar file action is not very common, so blocking them may not
> ST> be that bad for the users. Window of blocking can be reduced by
>
> ST> copying the jar files after the data segment. If some one thinks
> ST> making jar file requests fail while backup is running is better
> ST> approach, I am ok with it.
>
>
>
> >> - Testing: I think it is important to test that this handles
>
> >> rollback of transactions correctly. There should be a test
> >> that runs with a higher than normal frequency for
> >> transaction rollbacks. - Generally, tests should make
> >> sure that ACID properties are
>
> >> guaranteed after a restore.
> >>
>
>
> ST> Agree with you. I will add tests to cover the above scenarios.
>
>
> ST> Thanks
> ST> -suresht
>
>
>
>
>
>
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST(" == Suresh Thalamati (JIRA) <de...@db.apache.org> writes:
ST(> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
ST(> Suresh Thalamati updated DERBY-239:
ST(> -----------------------------------
ST(> Attachment: onlinebackup.html
ST(> functional/design spec based on the input I got on the derby-dev list.
Very good spec that seems to cover the necessary issues. I have a few
minor comments:
- Log files: I would think you may in some cases have to copy
log files from before the backup checkpoint in order to be
able to roll-back transactions that started before the
backup. (Or am I missing something).
- Jar files: What if a jar file is stored in the database
during backup? Will this be allowed? If yes, how is it
made sure that this is copied into the backup.
- Testing: I think it is important to test that this handles
rollback of transactions correctly. There should be a test
that runs with a higher than normal frequency for
transaction rollbacks.
- Generally, tests should make sure that ACID properties are
guaranteed after a restore.
--
Øystein
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup.html
functional/design spec based on the input I got on the derby-dev list.
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have reviewed and run tests on this patch. It has been committed as
326824
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_1.diff
>
>
> This patch adds some code necessary to support real-time online backup that
> does not block writes when database backup is in progress. All the current functional tests
> passed with these changes. It would be great if some can review and commit this patch.
>
> This patch changes the way data segment and log is backed up without blocking
> the writes.
>
> Data Segment Backup:
>
> o The containers to be backed up are found by scanning the files in seg0.
>
> o Each container is backed up by reading all the pages through the page cache,
> and then writing to the backup container. Pages are latched while
> writing into the backup container.
> o Not necessary to backup containers in any particular order , All updates
> that happens after a container is backed will be redone using the
> transaction log on restore.
>
> MT cases:
> 1) Each page is latched when it is written to the backup to prevent partial
> written pages sneaking into the backup.
> 2) Thread that is backing up the container will stop if another thread
> requests removal of the container when container is being backed up.
> 3) Truncate of the container blocks if the container is being backed up.
> 4) Partially created containers will not be backed up. Container cache will
> not return the container items until the creation is complete.
> (No changes are not for this case , that is how it currently works).
>
>
> Transaction Log Backup:
>
> Transaction Log file backup in two phases:
> 1) First Check point info and the the log files are backed up before the
> data segment.
> 2) After the data segment is backed up , all the log files
> that are generated after tha backup started are also copied into the backup.
>
> MT cases:
> 1) If there is a checkpoint in progress, backup will wait for the
> checkpoint to complete before copying checkpoint control information
> into the backup.
>
>
> Testing : All functional tests(derbyall suite) passed on jdk142/Windows XP.
>
> svn status:
> M java\engine\org\apache\derby\impl\store\raw\log\ReadOnly.java
> M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
> M java\engine\org\apache\derby\impl\store\raw\RawStore.java
> M java\engine\org\apache\derby\impl\store\raw\data\BasePage.java
> M java\engine\org\apache\derby\impl\store\raw\data\InputStreamContainer.jav
> a
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
>
> M java\engine\org\apache\derby\impl\store\raw\data\CachedPage.java
> M java\engine\org\apache\derby\impl\store\raw\data\FileContainer.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseContainer.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseContainerHandle.java
>
> M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
> M java\engine\org\apache\derby\iapi\store\raw\log\LogFactory.java
> M java\engine\org\apache\derby\iapi\store\raw\data\DataFactory.java
> M java\engine\org\apache\derby\iapi\store\raw\ContainerHandle.java
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup_1.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_1.diff
This patch adds some code necessary to support real-time online backup that
does not block writes when database backup is in progress. All the current functional tests
passed with these changes. It would be great if some can review and commit this patch.
This patch changes the way data segment and log is backed up without blocking
the writes.
Data Segment Backup:
o The containers to be backed up are found by scanning the files in seg0.
o Each container is backed up by reading all the pages through the page cache,
and then writing to the backup container. Pages are latched while
writing into the backup container.
o Not necessary to backup containers in any particular order , All updates
that happens after a container is backed will be redone using the
transaction log on restore.
MT cases:
1) Each page is latched when it is written to the backup to prevent partial
written pages sneaking into the backup.
2) Thread that is backing up the container will stop if another thread
requests removal of the container when container is being backed up.
3) Truncate of the container blocks if the container is being backed up.
4) Partially created containers will not be backed up. Container cache will
not return the container items until the creation is complete.
(No changes are not for this case , that is how it currently works).
Transaction Log Backup:
Transaction Log file backup in two phases:
1) First Check point info and the the log files are backed up before the
data segment.
2) After the data segment is backed up , all the log files
that are generated after tha backup started are also copied into the backup.
MT cases:
1) If there is a checkpoint in progress, backup will wait for the
checkpoint to complete before copying checkpoint control information
into the backup.
Testing : All functional tests(derbyall suite) passed on jdk142/Windows XP.
svn status:
M java\engine\org\apache\derby\impl\store\raw\log\ReadOnly.java
M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
M java\engine\org\apache\derby\impl\store\raw\RawStore.java
M java\engine\org\apache\derby\impl\store\raw\data\BasePage.java
M java\engine\org\apache\derby\impl\store\raw\data\InputStreamContainer.jav
a
M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M java\engine\org\apache\derby\impl\store\raw\data\CachedPage.java
M java\engine\org\apache\derby\impl\store\raw\data\FileContainer.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseContainer.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseContainerHandle.java
M java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
M java\engine\org\apache\derby\iapi\store\raw\log\LogFactory.java
M java\engine\org\apache\derby\iapi\store\raw\data\DataFactory.java
M java\engine\org\apache\derby\iapi\store\raw\ContainerHandle.java
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup_1.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: obtest_customer.jar
test jar file was missing from the earlier onlinebackup_6.diff patch. This jar file should be added to :
org/apache/derbyTesting/functionTests/tests/store/ directory along with the onlinebackup_6.diff patch.
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> I agree, at this point I think what you have implemented is a useful
MM> feature. It is much more online than 10.1 version. I agree with
MM> Oystein there may be ways to improve - but maybe those can be logged
MM> as new improvements.
I agree.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I agree, at this point I think what you have implemented is a useful
feature. It is much more online than 10.1 version. I agree with
Oystein there may be ways to improve - but maybe those can be logged
as new improvements.
Suresh Thalamati wrote:
> Øystein Grøvlen wrote:
>
>> Suresh,
>>
>> Thanks for considering my suggestions. A few responses below.
>>
>> --
>> Øystein
>>
>
>> >> * SYSCS_DISABLE_LOG_ARCHIVE_MODE()
>>
>> >> - Is checkBackupTransactionIsIdle() strictly necessary here? This
>> >> seems like an operation where failures could be handle at
>> >> statement level.
>> >>
>> ST> No. Idle check was necessary for issuing implicit commit
>> to be
>> ST> consistent with other backup calls.
>>
>> Is this really a backup call?
>>
>
> Not a real backup call, but it related to the backup functionality and
> disabling the log archive mode operation can not be rolled back, that
> was the reason behind doing the implicit commit for this call also. I
> will be removing the IDLE check and implicit commit in this case also
> along with other backup calls.
>
>
>> >>
>> java/engine/org/apache/derby/iapi/store/raw/xact/TransactionFactory.java
>>
>> >> * stopBackupBlockingOperations()
>>
>> >> - Name indicates that backup blocking operations are stopped, but
>> >> javadoc says that only new ones are blocked. I think the name
>> >> is misleading.
>> >> - Javadoc should be revisited for typos.
>>
>> ST> I can change them too :
>> ST> blockBackupBlockingOperations()
>> ST> unblockBackupBlockingOoperation()
>>
>> ST> Any suggestions ?
>>
>> I guess you mean unblockBackupBlockingOperations(). Names are OK. It
>> requires a bit of syntactic and semantic analysis to be grasp the
>> meaning of 'blockBackupBlocking', but I do not have a better
>> suggestion.
>>
>> >> java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
>>
>> >> * setBackupBlockingState()/setUnblockBackupState()
>>
>> >> - Names should be symmetric. (e.g., blockBackup/unblockBackup)
>> >> - Why do you have to wait for commit to unblock? Would it not be
>> >> sufficient to have completed the unlogged operations before
>> >> backup is started?
>>
>> ST> May be backup can be allowed once the unlogged operation is
>> complete,
>> ST> I have not looked in-depth to allow immediately after
>> the unlog
>> ST> operations is complete, one case that I thought likely to have
>>
>> ST> problem is JAR file removal on post commit conflicting with
>> backup
>> ST> jar copy operation.
>>
>> ST> For now, I think enforcing the COMMIT restriction is ok.
>>
>> Enforcing the commit restriction is the root of the discussions we
>> have had on idle transactions and automatic commit. If it is
>> necessary, things would be more straight-forward.
>> I would think whether the COMMIT restriction is necessary or not,
>> depends on whether one at restore will be able to roll back these
>> operations. I guess an index is ok to back up after it has been
>> created since it should be possible to roll back the creation by
>> dropping the index should the transaction never commit.
>>
>
> Commit/Rollback is not a necessary condition, technically backup needs
> to be blocked only for the duration of a unlogged operation. It is
> just simpler to test and implement online backup if backup is
> unblocked on commit/rollback in these cases. To unblock immediately
> after the unlogged operation, I would need to go through all the code
> for unlogged operation and make sure unblock of backup is done at the
> right place and testing becomes more tricky.
>
> I believe blocking the backup until unlogged operations are committed
> is acceptable for now, because typically users will issue a
> commit/rollback after the unlogged operations are complete. User
> application is required to do anything special in this case, backup
> will just block for more time than ideally required. If some one finds
> this is not acceptable, online backup can be enhanced to block only
> for the duration of the unlogged operation at a later point of time.
>
>
> Thanks
> -suresh
>
>
>
>
>
>
>
>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen wrote:
> Suresh,
>
> Thanks for considering my suggestions. A few responses below.
>
> --
> Øystein
>
> >> * SYSCS_DISABLE_LOG_ARCHIVE_MODE()
>
> >> - Is checkBackupTransactionIsIdle() strictly necessary here? This
> >> seems like an operation where failures could be handle at
> >> statement level.
> >>
>
> ST> No. Idle check was necessary for issuing implicit commit to be
> ST> consistent with other backup calls.
>
> Is this really a backup call?
>
Not a real backup call, but it related to the backup functionality and
disabling the log archive mode operation can not be rolled back, that
was the reason behind doing the implicit commit for this call also. I
will be removing the IDLE check and implicit commit in this case also
along with other backup calls.
> >> java/engine/org/apache/derby/iapi/store/raw/xact/TransactionFactory.java
>
> >> * stopBackupBlockingOperations()
>
> >> - Name indicates that backup blocking operations are stopped, but
> >> javadoc says that only new ones are blocked. I think the name
> >> is misleading.
> >> - Javadoc should be revisited for typos.
>
> ST> I can change them too :
> ST> blockBackupBlockingOperations()
> ST> unblockBackupBlockingOoperation()
>
> ST> Any suggestions ?
>
> I guess you mean unblockBackupBlockingOperations(). Names are OK. It
> requires a bit of syntactic and semantic analysis to be grasp the
> meaning of 'blockBackupBlocking', but I do not have a better
> suggestion.
>
> >> java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
>
> >> * setBackupBlockingState()/setUnblockBackupState()
>
> >> - Names should be symmetric. (e.g., blockBackup/unblockBackup)
> >> - Why do you have to wait for commit to unblock? Would it not be
> >> sufficient to have completed the unlogged operations before
> >> backup is started?
>
> ST> May be backup can be allowed once the unlogged operation is complete,
> ST> I have not looked in-depth to allow immediately after the unlog
> ST> operations is complete, one case that I thought likely to have
>
> ST> problem is JAR file removal on post commit conflicting with backup
> ST> jar copy operation.
>
> ST> For now, I think enforcing the COMMIT restriction is ok.
>
> Enforcing the commit restriction is the root of the discussions we
> have had on idle transactions and automatic commit. If it is
> necessary, things would be more straight-forward.
>
> I would think whether the COMMIT restriction is necessary or not,
> depends on whether one at restore will be able to roll back these
> operations. I guess an index is ok to back up after it has been
> created since it should be possible to roll back the creation by
> dropping the index should the transaction never commit.
>
Commit/Rollback is not a necessary condition, technically backup needs
to be blocked only for the duration of a unlogged operation. It is
just simpler to test and implement online backup if backup is
unblocked on commit/rollback in these cases. To unblock immediately
after the unlogged operation, I would need to go through all the code
for unlogged operation and make sure unblock of backup is done at the
right place and testing becomes more tricky.
I believe blocking the backup until unlogged operations are committed
is acceptable for now, because typically users will issue a
commit/rollback after the unlogged operations are complete. User
application is required to do anything special in this case, backup
will just block for more time than ideally required. If some one finds
this is not acceptable, online backup can be enhanced to block only
for the duration of the unlogged operation at a later point of time.
Thanks
-suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Suresh,
Thanks for considering my suggestions. A few responses below.
--
Øystein
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
>> * SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE()
>> - Same comment on ONLINE as above
>> - Could not boolean parameters be used now?
ST> I am not sure boolean paramets work with stored procedures now.
ST> Even if the support available, just to be consistent with existing
ST> backup procedure parameters types, I think using INT is better for the
ST> new procedures also.
I agree.
>> * backupDatabaseAndEnableLogArchiveMode()
>> - Most of the code here is the same as in backupDatabase().
>> (Another argument for pushing this code down to a lower layer.)
>> To avoid code duplication, whether to enable archiving could have
>> been a flag to backupDatabase.
ST> Added this function to support implicit commit/rollback, This function
ST> may not be needed any more. I will just call the low level backup
ST> calls directly.
Good.
>> * SYSCS_DISABLE_LOG_ARCHIVE_MODE()
>> - Is checkBackupTransactionIsIdle() strictly necessary here? This
>> seems like an operation where failures could be handle at
>> statement level.
>>
ST> No. Idle check was necessary for issuing implicit commit to be
ST> consistent with other backup calls.
Is this really a backup call?
>> java/engine/org/apache/derby/iapi/store/raw/xact/TransactionFactory.java
>> * stopBackupBlockingOperations()
>> - Name indicates that backup blocking operations are stopped, but
>> javadoc says that only new ones are blocked. I think the name
>> is misleading.
>> - Javadoc should be revisited for typos.
ST> I can change them too :
ST> blockBackupBlockingOperations()
ST> unblockBackupBlockingOoperation()
ST> Any suggestions ?
I guess you mean unblockBackupBlockingOperations(). Names are OK. It
requires a bit of syntactic and semantic analysis to be grasp the
meaning of 'blockBackupBlocking', but I do not have a better
suggestion.
>> java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
>> * setBackupBlockingState()/setUnblockBackupState()
>> - Names should be symmetric. (e.g., blockBackup/unblockBackup)
>> - Why do you have to wait for commit to unblock? Would it not be
>> sufficient to have completed the unlogged operations before
>> backup is started?
ST> May be backup can be allowed once the unlogged operation is complete,
ST> I have not looked in-depth to allow immediately after the unlog
ST> operations is complete, one case that I thought likely to have
ST> problem is JAR file removal on post commit conflicting with backup
ST> jar copy operation.
ST> For now, I think enforcing the COMMIT restriction is ok.
Enforcing the commit restriction is the root of the discussions we
have had on idle transactions and automatic commit. If it is
necessary, things would be more straight-forward.
I would think whether the COMMIT restriction is necessary or not,
depends on whether one at restore will be able to roll back these
operations. I guess an index is ok to back up after it has been
created since it should be possible to roll back the creation by
dropping the index should the transaction never commit.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Hi Øystein,
Thanks for reviewing the online backup patches, my comments are
in-line for the issues , i did not respond in my earlier e-mail.
Øystein Grøvlen (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361535 ]
>
> Øystein Grøvlen commented on DERBY-239:
> ---------------------------------------
>
> I have reviewed the latest patches (3 through6). (A bit late I must admit). Here are my comments:
>
> java/engine/org/apache/derby/catalog/SystemProcedures.java
>
> * SYSCS_BACKUP_DATABASE()
> - "By default this procedure will wait ..." Is it possible to
> change this behavior for this particular procedure? If not,
> "by default" is a bit misleading
> - "wait for the backup blocking unlogged operations to complete
> ..." is a bit heavy. I suggest just "wait for any unlogged
> operations to complete ..."
>
I will fix the comments.
> * SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE()
> - Same comment on ONLINE as above
> - Could not boolean parameters be used now?
I am not sure boolean paramets work with stored procedures now.
Even if the support available, just to be consistent with existing
backup procedure parameters types, I think using INT is better for the
new procedures also.
>
> * backupDatabaseAndEnableLogArchiveMode()
> - Most of the code here is the same as in backupDatabase().
> (Another argument for pushing this code down to a lower layer.)
> To avoid code duplication, whether to enable archiving could have
> been a flag to backupDatabase.
Added this function to support implicit commit/rollback, This function
may not be needed any more. I will just call the low level backup
calls directly.
>
> * SYSCS_DISABLE_LOG_ARCHIVE_MODE()
> - Is checkBackupTransactionIsIdle() strictly necessary here? This
> seems like an operation where failures could be handle at
> statement level.
>
No. Idle check was necessary for issuing implicit commit to be
consistent with other backup calls.
>
> java/engine/org/apache/derby/iapi/store/access/AccessFactory.java
>
> * backup(String ...)
> - "Please see cloudscape on line ..." Derby?
>
> * backup(File ...)
> - I will just remind you of DERBY-665. I could do that work, but
> I am not sure it is a good idea for other people change the
> backup code while you are working on it. (May create merge
> conflicts for you.)
Thanks for reminding , I will fix it.
>
>
> java/engine/org/apache/derby/iapi/store/raw/xact/RawTransaction.java
>
> * setBackupBlockingState()
> - I do not like the name for this method. I suggest calling it
> blockBackup() or something like that. At least, the javadoc
> should explain what is meant by "backup blocking state".
>
Thanks for the suggestion , I will change the names to blockBackup()
and unblockBackup() .
>
> java/engine/org/apache/derby/iapi/store/raw/xact/TransactionFactory.java
>
> * stopBackupBlockingOperations()
> - Name indicates that backup blocking operations are stopped, but
> javadoc says that only new ones are blocked. I think the name
> is misleading.
> - Javadoc should be revisited for typos.
I can change them too :
blockBackupBlockingOperations()
unblockBackupBlockingOoperation()
Any suggestions ?
>
>
> java/engine/org/apache/derby/impl/db/BasicDatabase.java
>
> * backupAndEnableLogArchiveMode()
> - Non-standard indentation for parameters?
Not sure why I did that, will fix it.
>
>
> java/engine/org/apache/derby/impl/store/raw/RawStore.java
>
> * backup(String, boolean)
> - Typo in comment: "Check if there any backup ..." Remove "there"?
>
> * backupAndEnableLogArchiveMode(String, boolean)
> - Why do you need a finally clause? Would not a catch clause be
> sufficient? Then, you could eliminate the local 'error'
> variable.
if I catch on Throwable , may be I don't need finally in this case.
>
>
> java/engine/org/apache/derby/impl/store/raw/data/RFResource.java
>
> * add()/remove()
> - Are the casts to RawTransaction safe? Does this assumption have
> any impact on the modularity of the code?
Yes. RawTransaction is the one that implements Transaction, and the
way the Transaction is acquired it is sure to be RawTransaction.
Having said that I don't like the casting either , will look around
and see if there is a simple way to get RawTransaction directly.
>
> * serviceImmediately()
> - How is this change related to backup?
If the jar files are not removed on COMMIT , removal work gets
assigned to RawStoreDaemon(postcommit) thread. Backup of jars and
Removal of Jar can conflict. Blocking the RawStoreDaemon on backup may
not be a good idea , so change the removal of jars to happen on the
commit calls itself and also I think it better remove files on commit.
>
>
> java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
>
> * setBackupBlockingState()/setUnblockBackupState()
> - Names should be symmetric. (e.g., blockBackup/unblockBackup)
> - Why do you have to wait for commit to unblock? Would it not be
> sufficient to have completed the unlogged operations before
> backup is started?
May be backup can be allowed once the unlogged operation is complete,
I have not looked in-depth to allow immediately after the unlog
operations is complete, one case that I thought likely to have
problem is JAR file removal on post commit conflicting with backup
jar copy operation.
For now, I think enforcing the COMMIT restriction is ok.
>
>
> java/engine/org/apache/derby/impl/store/raw/xact/XactFactory.java
>
> * canStartBackupBlockingOperation()
> - Since this method now may wait for backup to complete, I do not
> feel canStartBackupBlockingOperation() is a good name.
> - Symmetric naming with backupBlockingOperationFinished() would
> make it more evident that these two functions should be called
> in pairs.
will change these name also too :
blockBackup() and unblockBackup()
>
>
> java/engine/org/apache/derby/loc/messages_en.properties
>
> * XSRSA.S
> - Suggest the following change:
> "Cannot backup the database when unlogged operations are uncommitted. Please commit the transactions with backup blocking operations or use the backup procedure with option to wait for them to complete."
> - Generally, I am not very fond of these long error messages. I
> think it would be better with just a single sentence, and then
> the user should be able to look up an explanation in a manual.
>
> * XSRSB.S
> - Suggest the following change:
> "Backup operation can not be performed in an active transaction. Please start a new transaction to execute backup procedures."
>
Thanks for the suggestion , I will update the error messages.
>
> java/testing/org/apache/derbyTesting/functionTests/tests/store/OnlineBackupTest1.java
>
> * runTest()
> - Should have more than one unlogged operation in parallel in
> order to fully test that the counting of unlogged operations
> work.
> - Suggest to add test that when doing backup in a non-idle
> transaction, previous work has not been rolled back when backup
> fails, and that one can continue with more operations within the
> same transaction.
>
> * runConsistencyChecker()
> - This does only check consistnecy of internal structures. Should
> also check consistency of application data. Maybe you could
> execute a select?
>
> * performDmlActions()
> - I assume the intention here is to do "while (!stopActivity)"
I will enhance the test to address the above mentioned cases.
>
> * endUnloggedAction()
> - What is the role of the insert? Is is unlogged? It is not
> evident from the method name or Javadoc why you are doing
> inserts here.
I intended to check if that insert will make it to backup.
>
> * select()
> - What is the point of doing consistency checks on columns that
> are not updated? If id and name does not match, that must be
> caused by errors in code that is not particular to backup.
>
u are right, I will change the select to check on columns that are
being updated.
>
> java/testing/org/apache/derbyTesting/functionTests/tests/store/OnlineBackupTest3.java
>
> * installJarTest()
> - Typos in comment: "// followng backup call should because jar
> operation is pending". Should what?
> - Comment say: "//Now commit the jar operation in connection1 for
> backup to proceed." The next statement does an insert. This is
> confusing.
> - It would be nice if the test checked that jar operation is still
> waiting for backup when create index has completed, but I guess
> this is a bit difficult to achieve.
>
> * A mix of tab and spaces for indentation. For new files that
> should not be necessary!
>
> * removeJarTest()
> - Comment copied from installJarTest?
> "// wait for customer app jar installation to finish now. "
I will fix them.
>
>
> java/testing/org/apache/derbyTesting/functionTests/tests/store/onlineBackupTest2.sql
>
> * Do you have idea of how frequently it happens that backup thread
> has not been blocked yet when backupdir is created?
In this test,If backup is blocking correctly on unlogged operations
backupdir should not get created.
>How long a
> sleep would you need to be certain?
>
I checked with 1000 msec sleep time, test failed on my laptop if
backup is not blocking correctly for the unlogged operations to commit.
Test will not fail even if sleep() is removed , it is just to make
sure there is a chance for the test to actually enter the backup
blocking code. I can not be certain how much sleep time will make
sure that will happen, it will depend on the hardware and OS on which
the test is being run.
Thanks
-suresht
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Thanks for the input Øystein, Mike. Now I am also convinced approach
#2 is better. As this is functional issue, I think it is better to
resolve this issue now than leave it for future. I will work on
submitting the patch with the second approach.
Thanks
-suresht
Mike Matrigali wrote:
> approach #2 seems more "online", if it doesn't cause other problems it
> seems better. In reality I don't think most users will see any
> difference as it seems wierd to on purpose be doing other things in
> a transaction and then execute a backup. Documentation should be clear
> that Derby cannot back out a backup.
>
> Suresh Thalamati wrote:
>
>> Thanks for taking time to review. My comments are in-line for some
>> of the questions, I will respond to the other questions in another
>> e-mail.
>>
>> From you comments, one issue that we are uable to come to conclusion,
>> is what to do if a backup call is issued in a transaction that already
>> has a pending unlogged operations.
>>
>> 1) Current approach:
>>
>> a) permit backup call only in a non-idle transaction.
>> b) issue a implicit commit/rollback after the backup call is done.
>>
>> 2) Other approch is :
>>
>> a) prevent backup calls only when a transaction already has executed
>> unlogged operations.
>> b) Don't issue implicit commit/rollback after the backup
>>
>> I am ok with either of these approaches. I would like to know if
>> anyone prefers one approach over the other.
>>
>>
>>
>> Øystein Grøvlen (JIRA) wrote:
>>
>>> [
>>> http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361535
>>> ]
>>> Øystein Grøvlen commented on DERBY-239:
>>> ---------------------------------------
>>>
>>> I have reviewed the latest patches (3 through6). (A bit late I must
>>> admit). Here are my comments:
>>>
>>
>>>
>>> * SYSCS_ONLINE_BACKUP_DATABASE()
>>> - Since both backup procedures are ONLINE, it is a bit misleading
>>> to use this word to distinguish between the two backup
>>> procedures. I guess the main reason for choosing a new name is
>>> the extra parameter. In that case, I think it would be better
>>> to name the new procedure, SYSCS_BACKUP_DATABASE_NOWAIT and
>>> leave out the parameter.
>>> - The javadoc does not say what will happen if wait is 0. Will one
>>> get an exception if there is unlogged operations?
>>
>>
>>
>> I also think SYSCS_BACKUP_DATABASE_NOWAIT makes more sense, will
>> change the the procedure names with ONLINE to NO_WAIT, if no one else
>> has any objections.
>>
>> SYSCS_BACKUP_DATABASE_NOWAIT
>> SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NO_WAIT
>>
>>
>>>
>>> * backupDatabase()
>>> - Is this the right layer for checking that the transaction is
>>> idle and for doing rollback/commit the transaction? Since this
>>> is a requirement for the logic at lower layers to work
>>> correctly, not something that is done because it is the
>>> desirable behavior of the system procedure, I feel that this
>>> should be done at a lower layer.
>>
>>
>>
>> I think so, because it is better to commit/rollback at the jdbc layer
>> level than in rawstore, becuase if we add anything in language area
>> for backup , it will surprise us.
>>
>> in non-idle transaction error case backup should not commit/rollback,
>> that is why I pushed the check to jdbc level and also checking for
>> transaction is IDLE is already exposed to JDBC for some other code.
>>
>> Other approach was to make rawstore throw the error and cacth the
>> error in the jdbc layer and decide to issue commit/rollback.
>> I generally don't like catching exception and then deciding what to
>> based on the SQLState, so I decided to check for the transaction
>> idle state in SystemProcedures.java
>>
>>> - I know when we discussed this isssue earlier, I agreed that
>>> checking that the transaction is idle was a good solution.
>>> However, thinking a bit more about this, I think it would be
>>> better to fail the transaction when unlogged operations have
>>> been performed by the same transaction. That would limit it to
>>> those who actually need to be affected, and it would
>>> significantly reduce the probability of someone ever
>>> experiencing this problem.
>>
>>
>>
>> It is very rare that some user stating a backup in a non-idle
>> transaction. If you think there is a possibilty of users executing
>> backup in non-idle transactions, then we can change the backup to
>> check only for non-logged cases as us suggested. But if
>> we follow this approach then we can commit/rollback after the backup.
>>
>>
>>> - I am not very fond of automatic commits like this.
>>
>>
>>
>> I think it was decided in earlier e-mails that backups need not be
>> transactional. So it is good to issue commit becuase we are sure
>> transaction is in idle state when backup was started.
>>
>> If this is
>>
>>> necessary, I think it would be better to require that backup is
>>> performed in autocommit mode. Then the implications would be
>>> more evident to the users and not catch someone by surprise.
>>
>>
>>
>>
>> I think forcing for the backup to work only on autocommit is likely to
>> break existing applications that are doing backup in autocommit=false
>> mode.
>>
>>>
>>>> Need a online backup feature that does not block update
>>>> operations when online backup is in progress.
>>>> --------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> Key: DERBY-239
>>>> URL: http://issues.apache.org/jira/browse/DERBY-239
>>>> Project: Derby
>>>> Type: New Feature
>>>> Components: Store
>>>> Versions: 10.1.1.0
>>>> Reporter: Suresh Thalamati
>>>> Assignee: Suresh Thalamati
>>>> Attachments: obtest_customer.jar, onlinebackup.html,
>>>> onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff,
>>>> onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff,
>>>> onlinebackup_6.diff
>>>>
>>>> Currently Derby allows users to perfoms online backups using
>>>> SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup
>>>> is in progress, update operations are temporarily blocked, but read
>>>> operations can still proceed.
>>>> Blocking update operations can be real issue specifically in client
>>>> server environments, because user requests will be blocked for a
>>>> long time if a backup is in the progress on the server.
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
approach #2 seems more "online", if it doesn't cause other problems it
seems better. In reality I don't think most users will see any
difference as it seems wierd to on purpose be doing other things in
a transaction and then execute a backup. Documentation should be clear
that Derby cannot back out a backup.
Suresh Thalamati wrote:
> Thanks for taking time to review. My comments are in-line for some
> of the questions, I will respond to the other questions in another e-mail.
>
> From you comments, one issue that we are uable to come to conclusion,
> is what to do if a backup call is issued in a transaction that already
> has a pending unlogged operations.
>
> 1) Current approach:
>
> a) permit backup call only in a non-idle transaction.
> b) issue a implicit commit/rollback after the backup call is done.
>
> 2) Other approch is :
>
> a) prevent backup calls only when a transaction already has executed
> unlogged operations.
> b) Don't issue implicit commit/rollback after the backup
>
> I am ok with either of these approaches. I would like to know if anyone
> prefers one approach over the other.
>
>
>
> Øystein Grøvlen (JIRA) wrote:
>
>> [
>> http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361535
>> ]
>> Øystein Grøvlen commented on DERBY-239:
>> ---------------------------------------
>>
>> I have reviewed the latest patches (3 through6). (A bit late I must
>> admit). Here are my comments:
>>
>
>>
>> * SYSCS_ONLINE_BACKUP_DATABASE()
>> - Since both backup procedures are ONLINE, it is a bit misleading
>> to use this word to distinguish between the two backup
>> procedures. I guess the main reason for choosing a new name is
>> the extra parameter. In that case, I think it would be better
>> to name the new procedure, SYSCS_BACKUP_DATABASE_NOWAIT and
>> leave out the parameter.
>> - The javadoc does not say what will happen if wait is 0. Will one
>> get an exception if there is unlogged operations?
>
>
> I also think SYSCS_BACKUP_DATABASE_NOWAIT makes more sense, will
> change the the procedure names with ONLINE to NO_WAIT, if no one else
> has any objections.
>
> SYSCS_BACKUP_DATABASE_NOWAIT
> SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NO_WAIT
>
>
>>
>> * backupDatabase()
>> - Is this the right layer for checking that the transaction is
>> idle and for doing rollback/commit the transaction? Since this
>> is a requirement for the logic at lower layers to work
>> correctly, not something that is done because it is the
>> desirable behavior of the system procedure, I feel that this
>> should be done at a lower layer.
>
>
> I think so, because it is better to commit/rollback at the jdbc layer
> level than in rawstore, becuase if we add anything in language area for
> backup , it will surprise us.
>
> in non-idle transaction error case backup should not commit/rollback,
> that is why I pushed the check to jdbc level and also checking for
> transaction is IDLE is already exposed to JDBC for some other code.
>
> Other approach was to make rawstore throw the error and cacth the error
> in the jdbc layer and decide to issue commit/rollback.
> I generally don't like catching exception and then deciding what to
> based on the SQLState, so I decided to check for the transaction idle
> state in SystemProcedures.java
>
>> - I know when we discussed this isssue earlier, I agreed that
>> checking that the transaction is idle was a good solution.
>> However, thinking a bit more about this, I think it would be
>> better to fail the transaction when unlogged operations have
>> been performed by the same transaction. That would limit it to
>> those who actually need to be affected, and it would
>> significantly reduce the probability of someone ever
>> experiencing this problem.
>
>
> It is very rare that some user stating a backup in a non-idle
> transaction. If you think there is a possibilty of users executing
> backup in non-idle transactions, then we can change the backup to check
> only for non-logged cases as us suggested. But if
> we follow this approach then we can commit/rollback after the backup.
>
>
>> - I am not very fond of automatic commits like this.
>
>
> I think it was decided in earlier e-mails that backups need not be
> transactional. So it is good to issue commit becuase we are sure
> transaction is in idle state when backup was started.
>
> If this is
>
>> necessary, I think it would be better to require that backup is
>> performed in autocommit mode. Then the implications would be
>> more evident to the users and not catch someone by surprise.
>
>
>
> I think forcing for the backup to work only on autocommit is likely to
> break existing applications that are doing backup in autocommit=false mode.
>
>>
>>> Need a online backup feature that does not block update operations
>>> when online backup is in progress.
>>> --------------------------------------------------------------------------------------------------------
>>>
>>>
>>> Key: DERBY-239
>>> URL: http://issues.apache.org/jira/browse/DERBY-239
>>> Project: Derby
>>> Type: New Feature
>>> Components: Store
>>> Versions: 10.1.1.0
>>> Reporter: Suresh Thalamati
>>> Assignee: Suresh Thalamati
>>> Attachments: obtest_customer.jar, onlinebackup.html,
>>> onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff,
>>> onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff,
>>> onlinebackup_6.diff
>>>
>>> Currently Derby allows users to perfoms online backups using
>>> SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup
>>> is in progress, update operations are temporarily blocked, but read
>>> operations can still proceed.
>>> Blocking update operations can be real issue specifically in client
>>> server environments, because user requests will be blocked for a long
>>> time if a backup is in the progress on the server.
>>
>>
>>
>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen wrote:
>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>
>
> ST> Thanks for taking time to review. My comments are in-line for some
> ST> of the questions, I will respond to the other questions in another
> ST> e-mail.
>
>
> ST> From you comments, one issue that we are uable to come to conclusion,
> ST> is what to do if a backup call is issued in a transaction that already
> ST> has a pending unlogged operations.
>
>
> ST> 1) Current approach:
>
> ST> a) permit backup call only in a non-idle transaction.
> ST> b) issue a implicit commit/rollback after the backup call is done.
>
> ST> 2) Other approch is :
>
> ST> a) prevent backup calls only when a transaction already has executed
> ST> unlogged operations.
>
> ST> b) Don't issue implicit commit/rollback after the backup
>
> ST> I am ok with either of these approaches. I would like to know if
> ST> anyone prefers one approach over the other.
>
> I prefer 2) for two reasons:
> - 2a) will impact less users than 1a). I agree that not many will
> be hit by any of them, but it is possible that someone may think
> of reading or recording information in the database as part of
> doing a backup. I think it is much less likely that someone
> will combine an unlogged operation with backup.
> - I think we should if possible avoid exceptions to standard
> behavior. Implicit commit/rollback is an exception to standard
> behavior. Such exceptions require specific documentation and
> makes the product more complex to use. Users tend not to read
> such documentation. 1a) is less of a problem than 1b) since the
> user will get an error if they are not aware of the problem.
> 1b) will not necessarily cause an error, but the transactional
> behavior of an application may be different from what the user
> thinks it is.
>
> That said, one can not use unlimited resources in order to get the
> perfect solution. If 2) is much more work than 1), I see the argument
> for doing 1) now and just file a JIRA issue for the better solution.
>
> >> * backupDatabase()
>
> >> - Is this the right layer for checking that the transaction is
> >> idle and for doing rollback/commit the transaction? Since this
> >> is a requirement for the logic at lower layers to work
> >> correctly, not something that is done because it is the
> >> desirable behavior of the system procedure, I feel that this
> >> should be done at a lower layer.
>
> ST> I think so, because it is better to commit/rollback at the jdbc layer
> ST> level than in rawstore, becuase if we add anything in language area
> ST> for backup , it will surprise us.
>
> Surprise who? In what way?
>
> ST> in non-idle transaction error case backup should not commit/rollback,
> ST> that is why I pushed the check to jdbc level and also checking for
> ST> transaction is IDLE is already exposed to JDBC for some other code.
>
>
> ST> Other approach was to make rawstore throw the error and cacth the
> ST> error in the jdbc layer and decide to issue commit/rollback.
>
> I do not understand why you need to catch the error. If I understand
> you correctly you just said that commit/rollback should not be done if
> the transaction is non-idle.
>
My understanding is right layer to issue implicit commit/rollbacks is
JDBC layer than store, because the transaction backup procedure is
using is setup from the jdbc layer.
With that assumption if check for the transaction is IDLE is done in
the store layer backup methods then if it is not IDLE then that info
has to passed back all the way to JDBC layer, so that backup procdure
does not issue a commit/rollback. If use the exception mechanism to
do that , then I have to check for Error state before issuing a
rollback, which I normally don't like to do. That is all the reason
behind doing check for transacion is IDLE before calling backup
methods in store.
if implicit commit/rollbacks are not done , then I agree with you the
right place do such checks is in the store layer.
> ST> I generally don't like catching exception and then deciding what to
> ST> based on the SQLState, so I decided to check for the transaction idle
> ST> state in SystemProcedures.java
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> Thanks for taking time to review. My comments are in-line for some
ST> of the questions, I will respond to the other questions in another
ST> e-mail.
ST> From you comments, one issue that we are uable to come to conclusion,
ST> is what to do if a backup call is issued in a transaction that already
ST> has a pending unlogged operations.
ST> 1) Current approach:
ST> a) permit backup call only in a non-idle transaction.
ST> b) issue a implicit commit/rollback after the backup call is done.
ST> 2) Other approch is :
ST> a) prevent backup calls only when a transaction already has executed
ST> unlogged operations.
ST> b) Don't issue implicit commit/rollback after the backup
ST> I am ok with either of these approaches. I would like to know if
ST> anyone prefers one approach over the other.
I prefer 2) for two reasons:
- 2a) will impact less users than 1a). I agree that not many will
be hit by any of them, but it is possible that someone may think
of reading or recording information in the database as part of
doing a backup. I think it is much less likely that someone
will combine an unlogged operation with backup.
- I think we should if possible avoid exceptions to standard
behavior. Implicit commit/rollback is an exception to standard
behavior. Such exceptions require specific documentation and
makes the product more complex to use. Users tend not to read
such documentation. 1a) is less of a problem than 1b) since the
user will get an error if they are not aware of the problem.
1b) will not necessarily cause an error, but the transactional
behavior of an application may be different from what the user
thinks it is.
That said, one can not use unlimited resources in order to get the
perfect solution. If 2) is much more work than 1), I see the argument
for doing 1) now and just file a JIRA issue for the better solution.
>> * backupDatabase()
>> - Is this the right layer for checking that the transaction is
>> idle and for doing rollback/commit the transaction? Since this
>> is a requirement for the logic at lower layers to work
>> correctly, not something that is done because it is the
>> desirable behavior of the system procedure, I feel that this
>> should be done at a lower layer.
ST> I think so, because it is better to commit/rollback at the jdbc layer
ST> level than in rawstore, becuase if we add anything in language area
ST> for backup , it will surprise us.
Surprise who? In what way?
ST> in non-idle transaction error case backup should not commit/rollback,
ST> that is why I pushed the check to jdbc level and also checking for
ST> transaction is IDLE is already exposed to JDBC for some other code.
ST> Other approach was to make rawstore throw the error and cacth the
ST> error in the jdbc layer and decide to issue commit/rollback.
I do not understand why you need to catch the error. If I understand
you correctly you just said that commit/rollback should not be done if
the transaction is non-idle.
ST> I generally don't like catching exception and then deciding what to
ST> based on the SQLState, so I decided to check for the transaction idle
ST> state in SystemProcedures.java
>> - I know when we discussed this isssue earlier, I agreed that
>> checking that the transaction is idle was a good solution.
>> However, thinking a bit more about this, I think it would be
>> better to fail the transaction when unlogged operations have
>> been performed by the same transaction. That would limit it to
>> those who actually need to be affected, and it would
>> significantly reduce the probability of someone ever
>> experiencing this problem.
ST> It is very rare that some user stating a backup in a non-idle
ST> transaction. If you think there is a possibilty of users executing
ST> backup in non-idle transactions, then we can change the backup to
ST> check only for non-logged cases as us suggested. But if
ST> we follow this approach then we can commit/rollback after the backup.
As I say above, even if it is rare, it is much more likely than
combining unlogged operations and backup.
>> - I am not very fond of automatic commits like this.
ST> I think it was decided in earlier e-mails that backups need not be
ST> transactional. So it is good to issue commit becuase we are sure
ST> transaction is in idle state when backup was started.
ST> If this is
>> necessary, I think it would be better to require that backup is
>> performed in autocommit mode. Then the implications would be
>> more evident to the users and not catch someone by surprise.
ST> I think forcing for the backup to work only on autocommit is likely to
ST> break existing applications that are doing backup in autocommit=false
ST> mode.
Good point. I rest my case for autocommit mode.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Thanks for taking time to review. My comments are in-line for some
of the questions, I will respond to the other questions in another
e-mail.
From you comments, one issue that we are uable to come to conclusion,
is what to do if a backup call is issued in a transaction that already
has a pending unlogged operations.
1) Current approach:
a) permit backup call only in a non-idle transaction.
b) issue a implicit commit/rollback after the backup call is done.
2) Other approch is :
a) prevent backup calls only when a transaction already has executed
unlogged operations.
b) Don't issue implicit commit/rollback after the backup
I am ok with either of these approaches. I would like to know if
anyone prefers one approach over the other.
Øystein Grøvlen (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361535 ]
>
> Øystein Grøvlen commented on DERBY-239:
> ---------------------------------------
>
> I have reviewed the latest patches (3 through6). (A bit late I must admit). Here are my comments:
>
>
> * SYSCS_ONLINE_BACKUP_DATABASE()
> - Since both backup procedures are ONLINE, it is a bit misleading
> to use this word to distinguish between the two backup
> procedures. I guess the main reason for choosing a new name is
> the extra parameter. In that case, I think it would be better
> to name the new procedure, SYSCS_BACKUP_DATABASE_NOWAIT and
> leave out the parameter.
> - The javadoc does not say what will happen if wait is 0. Will one
> get an exception if there is unlogged operations?
I also think SYSCS_BACKUP_DATABASE_NOWAIT makes more sense, will
change the the procedure names with ONLINE to NO_WAIT, if no one else
has any objections.
SYSCS_BACKUP_DATABASE_NOWAIT
SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NO_WAIT
>
> * backupDatabase()
> - Is this the right layer for checking that the transaction is
> idle and for doing rollback/commit the transaction? Since this
> is a requirement for the logic at lower layers to work
> correctly, not something that is done because it is the
> desirable behavior of the system procedure, I feel that this
> should be done at a lower layer.
I think so, because it is better to commit/rollback at the jdbc layer
level than in rawstore, becuase if we add anything in language area
for backup , it will surprise us.
in non-idle transaction error case backup should not commit/rollback,
that is why I pushed the check to jdbc level and also checking for
transaction is IDLE is already exposed to JDBC for some other code.
Other approach was to make rawstore throw the error and cacth the
error in the jdbc layer and decide to issue commit/rollback.
I generally don't like catching exception and then deciding what to
based on the SQLState, so I decided to check for the transaction
idle state in SystemProcedures.java
> - I know when we discussed this isssue earlier, I agreed that
> checking that the transaction is idle was a good solution.
> However, thinking a bit more about this, I think it would be
> better to fail the transaction when unlogged operations have
> been performed by the same transaction. That would limit it to
> those who actually need to be affected, and it would
> significantly reduce the probability of someone ever
> experiencing this problem.
It is very rare that some user stating a backup in a non-idle
transaction. If you think there is a possibilty of users executing
backup in non-idle transactions, then we can change the backup to
check only for non-logged cases as us suggested. But if
we follow this approach then we can commit/rollback after the backup.
> - I am not very fond of automatic commits like this.
I think it was decided in earlier e-mails that backups need not be
transactional. So it is good to issue commit becuase we are sure
transaction is in idle state when backup was started.
If this is
> necessary, I think it would be better to require that backup is
> performed in autocommit mode. Then the implications would be
> more evident to the users and not catch someone by surprise.
I think forcing for the backup to work only on autocommit is likely to
break existing applications that are doing backup in autocommit=false
mode.
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361535 ]
Øystein Grøvlen commented on DERBY-239:
---------------------------------------
I have reviewed the latest patches (3 through6). (A bit late I must admit). Here are my comments:
java/engine/org/apache/derby/catalog/SystemProcedures.java
* SYSCS_BACKUP_DATABASE()
- "By default this procedure will wait ..." Is it possible to
change this behavior for this particular procedure? If not,
"by default" is a bit misleading
- "wait for the backup blocking unlogged operations to complete
..." is a bit heavy. I suggest just "wait for any unlogged
operations to complete ..."
* SYSCS_ONLINE_BACKUP_DATABASE()
- Since both backup procedures are ONLINE, it is a bit misleading
to use this word to distinguish between the two backup
procedures. I guess the main reason for choosing a new name is
the extra parameter. In that case, I think it would be better
to name the new procedure, SYSCS_BACKUP_DATABASE_NOWAIT and
leave out the parameter.
- The javadoc does not say what will happen if wait is 0. Will one
get an exception if there is unlogged operations?
* backupDatabase()
- Is this the right layer for checking that the transaction is
idle and for doing rollback/commit the transaction? Since this
is a requirement for the logic at lower layers to work
correctly, not something that is done because it is the
desirable behavior of the system procedure, I feel that this
should be done at a lower layer.
- I know when we discussed this isssue earlier, I agreed that
checking that the transaction is idle was a good solution.
However, thinking a bit more about this, I think it would be
better to fail the transaction when unlogged operations have
been performed by the same transaction. That would limit it to
those who actually need to be affected, and it would
significantly reduce the probability of someone ever
experiencing this problem.
- I am not very fond of automatic commits like this. If this is
necessary, I think it would be better to require that backup is
performed in autocommit mode. Then the implications would be
more evident to the users and not catch someone by surprise.
- The javadoc for the system procedures that use this function
should state the requirement imposed here (idle transaction) and
that the transaction will be committed if backup is succesful.
* SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE()
- Same comment on ONLINE as above
- Could not boolean parameters be used now?
* backupDatabaseAndEnableLogArchiveMode()
- Most of the code here is the same as in backupDatabase().
(Another argument for pushing this code down to a lower layer.)
To avoid code duplication, whether to enable archiving could have
been a flag to backupDatabase.
* SYSCS_DISABLE_LOG_ARCHIVE_MODE()
- Is checkBackupTransactionIsIdle() strictly necessary here? This
seems like an operation where failures could be handle at
statement level.
java/engine/org/apache/derby/iapi/store/access/AccessFactory.java
* backup(String ...)
- "Please see cloudscape on line ..." Derby?
* backup(File ...)
- I will just remind you of DERBY-665. I could do that work, but
I am not sure it is a good idea for other people change the
backup code while you are working on it. (May create merge
conflicts for you.)
java/engine/org/apache/derby/iapi/store/raw/xact/RawTransaction.java
* setBackupBlockingState()
- I do not like the name for this method. I suggest calling it
blockBackup() or something like that. At least, the javadoc
should explain what is meant by "backup blocking state".
java/engine/org/apache/derby/iapi/store/raw/xact/TransactionFactory.java
* stopBackupBlockingOperations()
- Name indicates that backup blocking operations are stopped, but
javadoc says that only new ones are blocked. I think the name
is misleading.
- Javadoc should be revisited for typos.
java/engine/org/apache/derby/impl/db/BasicDatabase.java
* backupAndEnableLogArchiveMode()
- Non-standard indentation for parameters?
java/engine/org/apache/derby/impl/store/raw/RawStore.java
* backup(String, boolean)
- Typo in comment: "Check if there any backup ..." Remove "there"?
* backupAndEnableLogArchiveMode(String, boolean)
- Why do you need a finally clause? Would not a catch clause be
sufficient? Then, you could eliminate the local 'error'
variable.
java/engine/org/apache/derby/impl/store/raw/data/RFResource.java
* add()/remove()
- Are the casts to RawTransaction safe? Does this assumption have
any impact on the modularity of the code?
* serviceImmediately()
- How is this change related to backup?
java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
* setBackupBlockingState()/setUnblockBackupState()
- Names should be symmetric. (e.g., blockBackup/unblockBackup)
- Why do you have to wait for commit to unblock? Would it not be
sufficient to have completed the unlogged operations before
backup is started?
java/engine/org/apache/derby/impl/store/raw/xact/XactFactory.java
* canStartBackupBlockingOperation()
- Since this method now may wait for backup to complete, I do not
feel canStartBackupBlockingOperation() is a good name.
- Symmetric naming with backupBlockingOperationFinished() would
make it more evident that these two functions should be called
in pairs.
java/engine/org/apache/derby/loc/messages_en.properties
* XSRSA.S
- Suggest the following change:
"Cannot backup the database when unlogged operations are uncommitted. Please commit the transactions with backup blocking operations or use the backup procedure with option to wait for them to complete."
- Generally, I am not very fond of these long error messages. I
think it would be better with just a single sentence, and then
the user should be able to look up an explanation in a manual.
* XSRSB.S
- Suggest the following change:
"Backup operation can not be performed in an active transaction. Please start a new transaction to execute backup procedures."
java/testing/org/apache/derbyTesting/functionTests/tests/store/OnlineBackupTest1.java
* runTest()
- Should have more than one unlogged operation in parallel in
order to fully test that the counting of unlogged operations
work.
- Suggest to add test that when doing backup in a non-idle
transaction, previous work has not been rolled back when backup
fails, and that one can continue with more operations within the
same transaction.
* runConsistencyChecker()
- This does only check consistnecy of internal structures. Should
also check consistency of application data. Maybe you could
execute a select?
* performDmlActions()
- I assume the intention here is to do "while (!stopActivity)"
* endUnloggedAction()
- What is the role of the insert? Is is unlogged? It is not
evident from the method name or Javadoc why you are doing
inserts here.
* select()
- What is the point of doing consistency checks on columns that
are not updated? If id and name does not match, that must be
caused by errors in code that is not particular to backup.
java/testing/org/apache/derbyTesting/functionTests/tests/store/OnlineBackupTest3.java
* installJarTest()
- Typos in comment: "// followng backup call should because jar
operation is pending". Should what?
- Comment say: "//Now commit the jar operation in connection1 for
backup to proceed." The next statement does an insert. This is
confusing.
- It would be nice if the test checked that jar operation is still
waiting for backup when create index has completed, but I guess
this is a bit difficult to achieve.
* A mix of tab and spaces for indentation. For new files that
should not be necessary!
* removeJarTest()
- Comment copied from installJarTest?
"// wait for customer app jar installation to finish now. "
java/testing/org/apache/derbyTesting/functionTests/tests/store/onlineBackupTest2.sql
* Do you have idea of how frequently it happens that backup thread
has not been blocked yet when backupdir is created? How long a
sleep would you need to be certain?
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup1.html
minor updates to the spec:
- Procedure can not be overloaded with different number of arguments in Derby. Added the keyword "ONLINE" to the new backup procedures to differentiate them from the old procedures.
-- jar file handling.
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Daniel John Debrunner (JIRA) wrote:
>
> Is the functional spec updated to reflect the new names of the procedures?
Not yet. I will update.
> I'm actually confused as to why the name now includes NOWAIT,
> not sure what the NOWAIT is meant to imply to the user of the procedures.
>To me it might seem that this would imply the procedure returns right
>away and the backup continues in the background.
hmm... did not think user could interpret NOWAIT this way too.
Following two new procedures, now includes NOWAIT:
SYSCS_BACKUP_DATABASE_NOWAIT(..)
SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NOWAIT(..).
These two procedures will throw error , if there are any uncommitted
unlogged operation before stating the backup. They will not wait for
the unlogged operations to complete unlike
SYSCS_BACKUP_DATABASE and
SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE
> Hopefully an updated functional spec will make it clear why NOWAIT is part of the name.
> For Derby to be easy to use the names of the system procedures should clearly indicate what they do.
I agree. Any suggestions ?
Other alternatives were :
SYSCS_ONLINE_BACKUP_DATABASE (... IN WAIT SMALLINT) and
SYSCS_ONLINE_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(...IN WAIT
SMALLINT)
users need to pass non-zero value to wait for unlogged operations ,
otherwise, procedures will throw error if there any unlogged operations.
Thanks
-suresht
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12363275 ]
Daniel John Debrunner commented on DERBY-239:
---------------------------------------------
Is the functional spec updated to reflect the new names of the procedures?
I'm actually confused as to why the name now includes NOWAIT, not sure what the NOWAIT is meant to imply to the user of the procedures. To me it might seem that this would imply the procedure returns right away and the backup continues in the background. Hopefully an updated functional spec will make it clear why NOWAIT is part of the name. For Derby to be easy to use the names of the system procedures should clearly indicate what they do.
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I am looking at reviewing/committing this patch.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_3.diff
>
> This patch adds code to support real-time online backup with unlogged
> operations. A consistent backup can not be made if there are pending
> transactions with unlogged operations or if unlogged operations
> occur when backup is in progress. Because container files can be copied
> to the backup before the transaction is committed and the data pages are
> flushed as part of the commit. As there is no transaction log for unlogged
> operations, while restoring from the backup database can not be restored to
> a consistent state. To make a consistent online backup in this scenario, this patch:
>
> 1) blocks online backup until all the transactions with unlogged operation are
> committed/aborted.
> 2) implicitly converts all unlogged operations to logged mode for the duration
> of the online backup, if they are started when backup is in progress.
>
> This patch also adds a test to test the online backup in parallel with some DML,
> DDL and unlogged operations.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
>
> svn stat:
>
> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\RawStore.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
>
> M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\TransactionFactory.java
> M java\testing\org\apache\derbyTesting\functionTests\tests\storetests\st_1.sql
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1_app.properties
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackup.java
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1.java
> M java\testing\org\apache\derbyTesting\functionTests\master\st_1.out
> A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest1.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Mike Matrigali wrote:
> I have tested this patch and reviewed the changes and
> have commited it as svn 345355.
>
> I have the following
> suggestions from review, and will look at any subsequent
> changes in these areas that Suresh would like to submit:
>
> o RawStore.java!canStartOnlineBackup() has a todo item - needs an
> exception error to be thrown.
Working on this one, tests needed some changes when exception is
thrown. Deferred it to fix it along with tests changes for the fix to
do commit/rollback after the backup procedures.
>
> o will things like sort and temporary containers cause online
> backup to wait if they are outstanding?
Backup should wait only for the real containers with non-logged operation.
Does the pre-existing
> backup cause sorts and temp tables to
> be logged during online backup?
> (these questions probably apply to BaseDataFileFactory.java)
>
Temp tables are not logged during the backup. I thought sort uses temp
tables, will check on this one.
Please let me know if you notice any of my changes are not doing the
above.
> o could you pick one place in the code to describe how all the
> routines work together to provide the functionality you need.
> I think basically put the description that you have in the
> JIRA for this patch somewhere in the code. The individual routines
> have comments but it hard to see how they all work together
> without one place descibing the interaction.
>
sure. I can do that in the main backup() routine in RawStore.java.
In future may be backup code in RawStore.java should moved into a
different BackupFactory class.
> o as I was reviewing I fixed some >80 line stuff.
> I know sometimes it is hard, but
> at least stuff like comments is really easy.
>
hmm.. Thanks for fixing them. I better add some check to my emacs to
avoid this in future.
> o may be interesting to add external sort testing and queries
> that cause temp tables. probably dependent on answer to above
> question.
>
I will add a test case to make sure backup is not blocked on temp tables.
Thanks
-suresh
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have tested this patch and reviewed the changes and
have commited it as svn 345355.
I have the following
suggestions from review, and will look at any subsequent
changes in these areas that Suresh would like to submit:
o RawStore.java!canStartOnlineBackup() has a todo item - needs an
exception error to be thrown.
o will things like sort and temporary containers cause online
backup to wait if they are outstanding? Does the pre-existing
backup cause sorts and temp tables to
be logged during online backup?
(these questions probably apply to BaseDataFileFactory.java)
o could you pick one place in the code to describe how all the
routines work together to provide the functionality you need.
I think basically put the description that you have in the
JIRA for this patch somewhere in the code. The individual routines
have comments but it hard to see how they all work together
without one place descibing the interaction.
o as I was reviewing I fixed some >80 line stuff.
I know sometimes it is hard, but
at least stuff like comments is really easy.
o may be interesting to add external sort testing and queries
that cause temp tables. probably dependent on answer to above
question.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_3.diff
>
> This patch adds code to support real-time online backup with unlogged
> operations. A consistent backup can not be made if there are pending
> transactions with unlogged operations or if unlogged operations
> occur when backup is in progress. Because container files can be copied
> to the backup before the transaction is committed and the data pages are
> flushed as part of the commit. As there is no transaction log for unlogged
> operations, while restoring from the backup database can not be restored to
> a consistent state. To make a consistent online backup in this scenario, this patch:
>
> 1) blocks online backup until all the transactions with unlogged operation are
> committed/aborted.
> 2) implicitly converts all unlogged operations to logged mode for the duration
> of the online backup, if they are started when backup is in progress.
>
> This patch also adds a test to test the online backup in parallel with some DML,
> DDL and unlogged operations.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
>
> svn stat:
>
> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\RawStore.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
>
> M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\TransactionFactory.java
> M java\testing\org\apache\derbyTesting\functionTests\tests\storetests\st_1.sql
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1_app.properties
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackup.java
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1.java
> M java\testing\org\apache\derbyTesting\functionTests\master\st_1.out
> A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest1.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_3.diff
This patch adds code to support real-time online backup with unlogged
operations. A consistent backup can not be made if there are pending
transactions with unlogged operations or if unlogged operations
occur when backup is in progress. Because container files can be copied
to the backup before the transaction is committed and the data pages are
flushed as part of the commit. As there is no transaction log for unlogged
operations, while restoring from the backup database can not be restored to
a consistent state. To make a consistent online backup in this scenario, this patch:
1) blocks online backup until all the transactions with unlogged operation are
committed/aborted.
2) implicitly converts all unlogged operations to logged mode for the duration
of the online backup, if they are started when backup is in progress.
This patch also adds a test to test the online backup in parallel with some DML,
DDL and unlogged operations.
TESTS : derbyall test suite passed on Windows XP/JDK142
It would be great if some can review and commit this patch.
svn stat:
M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
M java\engine\org\apache\derby\impl\store\raw\RawStore.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
M java\engine\org\apache\derby\iapi\store\raw\xact\TransactionFactory.java
M java\testing\org\apache\derbyTesting\functionTests\tests\storetests\st_1.sql
A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1_app.properties
A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackup.java
M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1.java
M java\testing\org\apache\derbyTesting\functionTests\master\st_1.out
A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest1.out
M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (DERBY-239) Need a online backup feature that does
not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati closed DERBY-239:
----------------------------------
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Fix For: 10.2.0.0
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup2.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff, onlinebackup_8.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-239) Need a online backup feature that does
not block update operations when online backup is in progress.
Posted by "Mike Matrigali (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Mike Matrigali updated DERBY-239:
---------------------------------
Description:
Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
backup is in the progress on the server.
was:
Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
backup is in the progress on the server.
Environment:
I committed the onlinebackup_8.diff patch to trunk as svn 373380
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff, onlinebackup_8.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I will look at reviewing/committing this change.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_4.diff
>
> This patch makes online backup call to wait/fail when unlogged operations like
> create index are pending. It also fixes derby-523 bug my making the existing
> log archive backup procedure to wait for the unlogged operation to complete.
>
> -- Two new procedures are added to allow the users to make backup wait/fail
> when unlogged operations are pending.
> -- prevents users starting backup in an non-idle transactions to avoid backup
> blocking forever if users starts backup in the same transaction as an
> unlogged operation.
> -- backup is not really transactional , to avoid any locking issues in the
> futures; backup procedures ends the transaction by implicitly doing commit
> when it is successful or rollback on any errors.
>
> A new backup test is added to store suite to test the above scenarios.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
> $ svn status
> M java\engine\org\apache\derby\impl\sql\catalog\DataDictionaryImpl.java
> M java\engine\org\apache\derby\impl\sql\catalog\DD_Version.java
> M java\engine\org\apache\derby\impl\db\BasicDatabase.java
> M java\engine\org\apache\derby\impl\store\access\RAMAccessManager.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\RawStore.java
> M java\engine\org\apache\derby\iapi\store\access\AccessFactory.java
> M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java
> M java\engine\org\apache\derby\iapi\reference\SQLState.java
> M java\engine\org\apache\derby\database\Database.java
> M java\engine\org\apache\derby\catalog\SystemProcedures.java
> M java\engine\org\apache\derby\loc\messages_en.properties
> M java\testing\org\apache\derbyTesting\functionTests\tests\storetests\st_1.sql
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2_app.properties
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2.sql
> M java\testing\org\apache\derbyTesting\functionTests\master\st_1.out
> A java\testing\org\apache\derbyTesting\functionTests\master\onlineBackupTest2.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
> M java\testing\org\apache\derbyTesting\functionTests\util\FTFileUtil.java
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have committed this patch as svn 354898 and 354901.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_4.diff
>
> This patch makes online backup call to wait/fail when unlogged operations like
> create index are pending. It also fixes derby-523 bug my making the existing
> log archive backup procedure to wait for the unlogged operation to complete.
>
> -- Two new procedures are added to allow the users to make backup wait/fail
> when unlogged operations are pending.
> -- prevents users starting backup in an non-idle transactions to avoid backup
> blocking forever if users starts backup in the same transaction as an
> unlogged operation.
> -- backup is not really transactional , to avoid any locking issues in the
> futures; backup procedures ends the transaction by implicitly doing commit
> when it is successful or rollback on any errors.
>
> A new backup test is added to store suite to test the above scenarios.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
> $ svn status
> M java\engine\org\apache\derby\impl\sql\catalog\DataDictionaryImpl.java
> M java\engine\org\apache\derby\impl\sql\catalog\DD_Version.java
> M java\engine\org\apache\derby\impl\db\BasicDatabase.java
> M java\engine\org\apache\derby\impl\store\access\RAMAccessManager.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\RawStore.java
> M java\engine\org\apache\derby\iapi\store\access\AccessFactory.java
> M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java
> M java\engine\org\apache\derby\iapi\reference\SQLState.java
> M java\engine\org\apache\derby\database\Database.java
> M java\engine\org\apache\derby\catalog\SystemProcedures.java
> M java\engine\org\apache\derby\loc\messages_en.properties
> M java\testing\org\apache\derbyTesting\functionTests\tests\storetests\st_1.sql
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2_app.properties
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2.sql
> M java\testing\org\apache\derbyTesting\functionTests\master\st_1.out
> A java\testing\org\apache\derbyTesting\functionTests\master\onlineBackupTest2.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
> M java\testing\org\apache\derbyTesting\functionTests\util\FTFileUtil.java
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_4.diff
This patch makes online backup call to wait/fail when unlogged operations like
create index are pending. It also fixes derby-523 bug my making the existing
log archive backup procedure to wait for the unlogged operation to complete.
-- Two new procedures are added to allow the users to make backup wait/fail
when unlogged operations are pending.
-- prevents users starting backup in an non-idle transactions to avoid backup
blocking forever if users starts backup in the same transaction as an
unlogged operation.
-- backup is not really transactional , to avoid any locking issues in the
futures; backup procedures ends the transaction by implicitly doing commit
when it is successful or rollback on any errors.
A new backup test is added to store suite to test the above scenarios.
TESTS : derbyall test suite passed on Windows XP/JDK142
It would be great if some can review and commit this patch.
$ svn status
M java\engine\org\apache\derby\impl\sql\catalog\DataDictionaryImpl.java
M java\engine\org\apache\derby\impl\sql\catalog\DD_Version.java
M java\engine\org\apache\derby\impl\db\BasicDatabase.java
M java\engine\org\apache\derby\impl\store\access\RAMAccessManager.java
M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
M java\engine\org\apache\derby\impl\store\raw\RawStore.java
M java\engine\org\apache\derby\iapi\store\access\AccessFactory.java
M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java
M java\engine\org\apache\derby\iapi\reference\SQLState.java
M java\engine\org\apache\derby\database\Database.java
M java\engine\org\apache\derby\catalog\SystemProcedures.java
M java\engine\org\apache\derby\loc\messages_en.properties
M java\testing\org\apache\derbyTesting\functionTests\tests\storetests\st_1.sql
A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2_app.properties
M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2.sql
M java\testing\org\apache\derbyTesting\functionTests\master\st_1.out
A java\testing\org\apache\derbyTesting\functionTests\master\onlineBackupTest2.out
M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
M java\testing\org\apache\derbyTesting\functionTests\util\FTFileUtil.java
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-239) Need a online backup feature that does
not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup2.html
minor updated to the spec. Changed the names of the new procedures to :
SYSCS_BACKUP_DATABASE_NOWAIT
SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE_NOWAIT
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup2.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff, onlinebackup_8.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati reassigned DERBY-239:
--------------------------------------
Assign To: Suresh Thalamati
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.0.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
I agree that option 1 is to be preferrable. I do not think there is
any need for transactional behavior for backup.
--
Øystein
>>>>> "ST(" == Suresh Thalamati (JIRA) <de...@db.apache.org> writes:
ST(> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12356240 ]
ST(> Suresh Thalamati commented on DERBY-239:
ST(> ----------------------------------------
ST(> What to do if backup is started in a transaction that already has unlogged operations executed?
ST(> In previous discussions about online backup, it was concluded that existing
ST(> backup procedures calls will WAIT for the transaction with unlogged
ST(> operations to commit before proceeding with the backup. One issue that was
ST(> missing from the discussion was, what to do if user starts a backup
ST(> in the same transaction that has unlogged operations executed before the backup call.
ST(> WAIT will not be an acceptable option here, because backup call will wait forever.
ST(> I can think of two ways this issue can be addressed:
ST(> 1) Add a restriction that backup procedures can only be called in a brand
ST(> NEW transaction. And also implicitly commit the backup transaction at the end of the
ST(> backup. Commit is not required as such to solve this problem, but it would
ST(> be cleaner because backup itself is not a rollback-able operation.
ST(> 2) Make backup procedures fail, if transaction that it is started in
ST(> contains unlogged operations.
ST(> I am inclined towards implementing the first option. Any comments/suggestion
ST(> will be appreciated.
ST(> Thanks
ST(> -suresht
>> Need a online backup feature that does not block update operations when online backup is in progress.
>> --------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup_1.diff
>>
>> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>> backup is in the progress on the server.
ST(> --
ST(> This message is automatically generated by JIRA.
ST(> -
ST(> If you think it was sent incorrectly contact one of the administrators:
ST(> http://issues.apache.org/jira/secure/Administrators.jspa
ST(> -
ST(> For more information on JIRA, see:
ST(> http://www.atlassian.com/software/jira
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
In general I also don't like implicit commits either.
I think there
is really only 2 options - possibly throw an error if the backup
is started by a thread with an outstanding unlogged operation or not. I
think the set lock mode command is
the only current one in Derby to do implicit commits, and I believe that
is part of the standard.
I don't think the nested transaction helps here as the problem is that
backup needs to wait for transactions with unlogged operations to
complete, which in the
case suresh is discussing the unlogged operation was performed by the
same thread doing the backup.
Just starting a nested transaction doesn't help, as the unlogged
operation is still out there, and will wait forever on itself.
Øystein Grøvlen wrote:
>>>>>>"MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
>
>
> MM> I like option 1, make sure it is well documented. I actually lean
> MM> toward even stronger, have the command commit the current transaction
> MM> before and after the backup.
>
> Generally, I do not like such implicit commits. It is likely to catch
> someone by surprise. On the other, I do not think many people would
> intentionally do backup as part of a larger transaction.
>
> In my opinion, the ideal solution would be to execute backup in a
> nested transaction. I do not know whether it is worth the effort.
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Thanks for the input. I will go with the option 1.
Nested transaction may not solve this problem , Nested Transaction
will be be on the same thread as the user thread, so user will not be
able to commit the unlogged operations for the backup operation to
proceed.
Thanks
-suresht
Øystein Grøvlen wrote:
>>>>>>"MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
>
>
> MM> I like option 1, make sure it is well documented. I actually lean
> MM> toward even stronger, have the command commit the current transaction
> MM> before and after the backup.
>
> Generally, I do not like such implicit commits. It is likely to catch
> someone by surprise. On the other, I do not think many people would
> intentionally do backup as part of a larger transaction.
>
> In my opinion, the ideal solution would be to execute backup in a
> nested transaction. I do not know whether it is worth the effort.
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> I like option 1, make sure it is well documented. I actually lean
MM> toward even stronger, have the command commit the current transaction
MM> before and after the backup.
Generally, I do not like such implicit commits. It is likely to catch
someone by surprise. On the other, I do not think many people would
intentionally do backup as part of a larger transaction.
In my opinion, the ideal solution would be to execute backup in a
nested transaction. I do not know whether it is worth the effort.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I like option 1, make sure it is well documented. I actually lean
toward even stronger, have the command commit the current transaction
before and after the backup.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12356240 ]
>
> Suresh Thalamati commented on DERBY-239:
> ----------------------------------------
>
> What to do if backup is started in a transaction that already has unlogged operations executed?
>
> In previous discussions about online backup, it was concluded that existing
> backup procedures calls will WAIT for the transaction with unlogged
> operations to commit before proceeding with the backup. One issue that was
> missing from the discussion was, what to do if user starts a backup
> in the same transaction that has unlogged operations executed before the backup call.
> WAIT will not be an acceptable option here, because backup call will wait forever.
>
> I can think of two ways this issue can be addressed:
>
> 1) Add a restriction that backup procedures can only be called in a brand
> NEW transaction. And also implicitly commit the backup transaction at the end of the
> backup. Commit is not required as such to solve this problem, but it would
> be cleaner because backup itself is not a rollback-able operation.
>
> 2) Make backup procedures fail, if transaction that it is started in
> contains unlogged operations.
>
>
> I am inclined towards implementing the first option. Any comments/suggestion
> will be appreciated.
>
>
> Thanks
> -suresht
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup_1.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12356240 ]
Suresh Thalamati commented on DERBY-239:
----------------------------------------
What to do if backup is started in a transaction that already has unlogged operations executed?
In previous discussions about online backup, it was concluded that existing
backup procedures calls will WAIT for the transaction with unlogged
operations to commit before proceeding with the backup. One issue that was
missing from the discussion was, what to do if user starts a backup
in the same transaction that has unlogged operations executed before the backup call.
WAIT will not be an acceptable option here, because backup call will wait forever.
I can think of two ways this issue can be addressed:
1) Add a restriction that backup procedures can only be called in a brand
NEW transaction. And also implicitly commit the backup transaction at the end of the
backup. Commit is not required as such to solve this problem, but it would
be cleaner because backup itself is not a rollback-able operation.
2) Make backup procedures fail, if transaction that it is started in
contains unlogged operations.
I am inclined towards implementing the first option. Any comments/suggestion
will be appreciated.
Thanks
-suresht
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup_1.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have run tests and committed this patch as svn 370530
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_7.diff
>
> This patch addresses the issues raised by Øystein in his review of previous
> online backup patches 3-6.
>
> - changed the backup procedures names with ONLINE to NOWAIT
> - removed the transaction Idle restriction to run backup procedures.
> - removed implicit commit/rollbacks.
> - Added a new lesser impact restriction, which only disallows backup call only if
> there are unlogged operations executed in the same transaction before the backup.
> - Removed casting to RawTransaction.
> - fixed Names and Comments.
> - Enhanced the tests with addional test cases suggested by Øystein.
>
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
>
> svn stat:
> M java\engine\org\apache\derby\impl\sql\catalog\DataDictionaryImpl.java
> M java\engine\org\apache\derby\impl\db\BasicDatabase.java
> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\RawStore.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
>
> M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
> M java\engine\org\apache\derby\iapi\store\access\AccessFactory.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\TransactionFactory.java
> M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java
> M java\engine\org\apache\derby\iapi\reference\SQLState.java
> M java\engine\org\apache\derby\catalog\SystemProcedures.java
> M java\engine\org\apache\derby\loc\messages_en.properties
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1.java
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2.sql
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
> M java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest1.out
> M java\testing\org\apache\derbyTesting\functionTests\master\onlineBackupTest2.out
> M java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does
not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_7.diff
This patch addresses the issues raised by Øystein in his review of previous
online backup patches 3-6.
- changed the backup procedures names with ONLINE to NOWAIT
- removed the transaction Idle restriction to run backup procedures.
- removed implicit commit/rollbacks.
- Added a new lesser impact restriction, which only disallows backup call only if
there are unlogged operations executed in the same transaction before the backup.
- Removed casting to RawTransaction.
- fixed Names and Comments.
- Enhanced the tests with addional test cases suggested by Øystein.
TESTS : derbyall test suite passed on Windows XP/JDK142
It would be great if some can review and commit this patch.
svn stat:
M java\engine\org\apache\derby\impl\sql\catalog\DataDictionaryImpl.java
M java\engine\org\apache\derby\impl\db\BasicDatabase.java
M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
M java\engine\org\apache\derby\impl\store\raw\RawStore.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
M java\engine\org\apache\derby\iapi\store\access\AccessFactory.java
M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
M java\engine\org\apache\derby\iapi\store\raw\xact\TransactionFactory.java
M java\engine\org\apache\derby\iapi\store\raw\RawStoreFactory.java
M java\engine\org\apache\derby\iapi\reference\SQLState.java
M java\engine\org\apache\derby\catalog\SystemProcedures.java
M java\engine\org\apache\derby\loc\messages_en.properties
M java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest1.java
M java\testing\org\apache\derbyTesting\functionTests\tests\store\onlineBackupTest2.sql
M java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
M java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest1.out
M java\testing\org\apache\derbyTesting\functionTests\master\onlineBackupTest2.out
M java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff, onlinebackup_7.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I am looking at reviewing/committing this.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_6.diff
>
> This patch adds code to support online backup when jar operations
> are running parallel to the backup. Jar files are not logged, but the
> system catalogs updates are logged when a jar file is added/replaced.
> If the jar file operations are allowed during the backup, system catalog
> (sys.sysfiles) table in the backup database can have a reference to a jar file
> that does not exist in the backup database. And also backup can contain
> partial written jar files. To make a consistent online backup, this patch:
>
> 1) Makes Backup operation wait/fail for all the jar operations activity in progress to complete.
> 2) Blocks jar file operations when a backup is in progress.
>
> This patch also adds a new test to test the online backup
> with jar operations.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
>
> svn stat:
>
> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
> M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\obtest_customer.jar
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3_app.properties
> A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have committed this patch (added the later attached .jar file by hand
to the commit). svn Committed revision 357269.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_6.diff
>
> This patch adds code to support online backup when jar operations
> are running parallel to the backup. Jar files are not logged, but the
> system catalogs updates are logged when a jar file is added/replaced.
> If the jar file operations are allowed during the backup, system catalog
> (sys.sysfiles) table in the backup database can have a reference to a jar file
> that does not exist in the backup database. And also backup can contain
> partial written jar files. To make a consistent online backup, this patch:
>
> 1) Makes Backup operation wait/fail for all the jar operations activity in progress to complete.
> 2) Blocks jar file operations when a backup is in progress.
>
> This patch also adds a new test to test the online backup
> with jar operations.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
>
> svn stat:
>
> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
> M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\obtest_customer.jar
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3_app.properties
> A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
I will add it to the Jira entry, did not realize that svn diff will
not pick it.
sorry
-suresh
Mike Matrigali wrote:
> looks good, but it looks like it is missing a file - maybe svn diff did
> not pick up a binary file. From running the tests it looks like I need
> a obtest_customer.jar for the new test.
>
> Suresh Thalamati (JIRA) wrote:
>
>> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>>
>> Suresh Thalamati updated DERBY-239:
>> -----------------------------------
>>
>> Attachment: onlinebackup_6.diff
>>
>> This patch adds code to support online backup when jar operations are
>> running parallel to the backup. Jar files are not logged, but the
>> system catalogs updates are logged when a jar file is added/replaced.
>> If the jar file operations are allowed during the backup, system
>> catalog (sys.sysfiles) table in the backup database can have a
>> reference to a jar file that does not exist in the backup database.
>> And also backup can contain partial written jar files. To make a
>> consistent online backup, this patch:
>>
>> 1) Makes Backup operation wait/fail for all the jar operations
>> activity in progress to complete.
>> 2) Blocks jar file operations when a backup is in progress.
>>
>> This patch also adds a new test to test the online backup with jar
>> operations.
>>
>> TESTS : derbyall test suite passed on Windows XP/JDK142
>>
>> It would be great if some can review and commit this patch.
>>
>> svn stat:
>>
>> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
>> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
>> M
>> java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
>> M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
>> M
>> java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
>> A
>> java\testing\org\apache\derbyTesting\functionTests\tests\store\obtest_customer.jar
>>
>> M
>> java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
>>
>> A
>> java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
>>
>> A
>> java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3_app.properties
>>
>> A
>> java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
>>
>> M
>> java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
>>
>>
>>
>>> Need a online backup feature that does not block update operations
>>> when online backup is in progress.
>>> --------------------------------------------------------------------------------------------------------
>>>
>>>
>>> Key: DERBY-239
>>> URL: http://issues.apache.org/jira/browse/DERBY-239
>>> Project: Derby
>>> Type: New Feature
>>> Components: Store
>>> Versions: 10.1.1.0
>>> Reporter: Suresh Thalamati
>>> Assignee: Suresh Thalamati
>>> Attachments: onlinebackup.html, onlinebackup1.html,
>>> onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff,
>>> onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>>
>>> Currently Derby allows users to perfoms online backups using
>>> SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup
>>> is in progress, update operations are temporarily blocked, but read
>>> operations can still proceed.
>>> Blocking update operations can be real issue specifically in client
>>> server environments, because user requests will be blocked for a long
>>> time if a backup is in the progress on the server.
>>
>>
>>
>
>
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
looks good, but it looks like it is missing a file - maybe svn diff did
not pick up a binary file. From running the tests it looks like I need
a obtest_customer.jar for the new test.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_6.diff
>
> This patch adds code to support online backup when jar operations
> are running parallel to the backup. Jar files are not logged, but the
> system catalogs updates are logged when a jar file is added/replaced.
> If the jar file operations are allowed during the backup, system catalog
> (sys.sysfiles) table in the backup database can have a reference to a jar file
> that does not exist in the backup database. And also backup can contain
> partial written jar files. To make a consistent online backup, this patch:
>
> 1) Makes Backup operation wait/fail for all the jar operations activity in progress to complete.
> 2) Blocks jar file operations when a backup is in progress.
>
> This patch also adds a new test to test the online backup
> with jar operations.
>
> TESTS : derbyall test suite passed on Windows XP/JDK142
>
> It would be great if some can review and commit this patch.
>
>
> svn stat:
>
> M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
> M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
> M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
> M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
> M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\obtest_customer.jar
> M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
> A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3_app.properties
> A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
> M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_6.diff
This patch adds code to support online backup when jar operations
are running parallel to the backup. Jar files are not logged, but the
system catalogs updates are logged when a jar file is added/replaced.
If the jar file operations are allowed during the backup, system catalog
(sys.sysfiles) table in the backup database can have a reference to a jar file
that does not exist in the backup database. And also backup can contain
partial written jar files. To make a consistent online backup, this patch:
1) Makes Backup operation wait/fail for all the jar operations activity in progress to complete.
2) Blocks jar file operations when a backup is in progress.
This patch also adds a new test to test the online backup
with jar operations.
TESTS : derbyall test suite passed on Windows XP/JDK142
It would be great if some can review and commit this patch.
svn stat:
M java\engine\org\apache\derby\impl\store\raw\xact\Xact.java
M java\engine\org\apache\derby\impl\store\raw\xact\XactFactory.java
M java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M java\engine\org\apache\derby\impl\store\raw\data\RFResource.java
M java\engine\org\apache\derby\iapi\store\raw\xact\RawTransaction.java
A java\testing\org\apache\derbyTesting\functionTests\tests\store\obtest_customer.jar
M java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3.java
A java\testing\org\apache\derbyTesting\functionTests\tests\store\OnlineBackupTest3_app.properties
A java\testing\org\apache\derbyTesting\functionTests\master\OnlineBackupTest3.out
M java\testing\org\apache\derbyTesting\functionTests\suites\storemore.runall
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
i have committed this patch as svn 354523. Please let the list know if
you see any problems on other jvms/OS/machines after getting this patch.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_5.diff
>
> This pacth fixes store/onlineBackupTest1.java failure on non-windows envirorment. Problem was unlogged operations thread and insert thread are working on the same connection. Test was failing becuase insert thread was committing the unlogged operation that was suppose to block the backup.
> This pacth modified the test , so that these threads works on different conenctions,
>
>
> Test passed on Windows XP and Linuix.
>
> It would be great if some one can commit this patch.
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_5.diff
This pacth fixes store/onlineBackupTest1.java failure on non-windows envirorment. Problem was unlogged operations thread and insert thread are working on the same connection. Test was failing becuase insert thread was committing the unlogged operation that was suppose to block the backup.
This pacth modified the test , so that these threads works on different conenctions,
Test passed on Windows XP and Linuix.
It would be great if some one can commit this patch.
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
sounds good to me. In place compress, at least currently is an ad-hoc
infrequent operation, and unlikely to be going on when a backup is
taking place.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361686 ]
>
> Suresh Thalamati commented on DERBY-239:
> ----------------------------------------
>
> inplace-compress with online backup problem:
>
> I was scanning through the code to find out any issues with online backup and in-place compress and came across the following code that does a checkpoint before truncating the container.
>
> FileContainer.java:
> protected void compressContainer(
> ....
> // make sure we don't execute redo recovery on any page
> // which is getting truncated. At this point we have an exclusive
> // table lock on the table, so after checkpoint no page change
> // can happen between checkpoint log record and compress of space.
> dataFactory.getRawStoreFactory().checkpoint();
>
>
> Above code assumes that redo will only start after the checkpoint
> done by the compress, that is true in crash-recovery. But
> restore from backup can start redo from a checkpoint that is taken when backup was started, which can be before the checkpoint done by compress. if compress is run in parallel to the backup, restore from the backup can FAIL because it can not find the pages needed by the redo if a container gets backed up after it is truncate by the compress.
>
> I could not think of an easy way to avoid the need for the compress to perform checkpoint while truncating a container.
>
> One way to make good online backup when in-place compress is
> in progress seems to be by enforcing the following restrictions similar to the way unlogged operations are handled:
>
> 1) Block in-place compress operation if backup is in progress and
> 2) Make backup operation wait/fail until compress is done.
>
>
> I don't like to add restrictions, but I guess compress is an infrequent operation, so it may be ok.
>
>
> Any comments/suggestions ?
>
> Thanks
> -suresht
>
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12361686 ]
Suresh Thalamati commented on DERBY-239:
----------------------------------------
inplace-compress with online backup problem:
I was scanning through the code to find out any issues with online backup and in-place compress and came across the following code that does a checkpoint before truncating the container.
FileContainer.java:
protected void compressContainer(
....
// make sure we don't execute redo recovery on any page
// which is getting truncated. At this point we have an exclusive
// table lock on the table, so after checkpoint no page change
// can happen between checkpoint log record and compress of space.
dataFactory.getRawStoreFactory().checkpoint();
Above code assumes that redo will only start after the checkpoint
done by the compress, that is true in crash-recovery. But
restore from backup can start redo from a checkpoint that is taken when backup was started, which can be before the checkpoint done by compress. if compress is run in parallel to the backup, restore from the backup can FAIL because it can not find the pages needed by the redo if a container gets backed up after it is truncate by the compress.
I could not think of an easy way to avoid the need for the compress to perform checkpoint while truncating a container.
One way to make good online backup when in-place compress is
in progress seems to be by enforcing the following restrictions similar to the way unlogged operations are handled:
1) Block in-place compress operation if backup is in progress and
2) Make backup operation wait/fail until compress is done.
I don't like to add restrictions, but I guess compress is an infrequent operation, so it may be ok.
Any comments/suggestions ?
Thanks
-suresht
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: obtest_customer.jar, onlinebackup.html, onlinebackup1.html, onlinebackup_1.diff, onlinebackup_2.diff, onlinebackup_3.diff, onlinebackup_4.diff, onlinebackup_5.diff, onlinebackup_6.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Daniel John Debrunner wrote:
>Suresh Thalamati (JIRA) wrote:
>
>
>
>> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12316434 ]
>>
>>Suresh Thalamati commented on DERBY-239:
>>----------------------------------------
>>
>>
>
>[comments just on this issue]
>
>
>
>>b) read each pages in to the page cache first and then latch the
>> page in the cache until a temporary copy of it is made. This approach
>> does not have extra overhead of extra latches on the page keys during writes , but
>> will pollute the page cache with the pages that are only required by the
>> backup; this might have impact on user operations because active user pages may
>> have been replaced by the backup pages in the page cache.
>>
>>or
>>
>>c) read pages into buffer pool and latch them while making a copy similar to
>>the above approach, but some how make sure that user pages are not kicked out
>>of the buffer pool.
>>
>>
>
>I think b) is the correct approach, but what is 'buffer pool' in c)?
>
>
>
>I think modifications to the cache would be useful for b), so that
>entries in the cache (through generic apis, not specific to store) could
>mark how "useful/valuable" they are. Just a simple scheme, lower numbers
>less valuable, higher numbers more valuable, and if it makes it easier
>to fix a range, e.g. 0-100, then that would be ok. Then the store could
>added pages to the cache with this weighting, e.g. (to get the general idea)
>
> pages for backup - weight 0
> overflow column pages - weight 10
> regular pages - weight 20
> leaf index pages - weight 30
> root index pages 80
>
>This weight would then be factored into the decision to throw pages out
>or not.
>
>This project could be independent of the online backup and could have
>benfits elsewhere.
>
>Dan.
>
>
>
>
>
"buffer pool" is hangover from my old job, I meant "page cache" in
option c). I also think b) is the correct approach,
if the cache can be enhanced as you described.
Thanks
-suresht
Re: page replacement [jira] Commented: (DERBY-239) Need a online
backup feature that does not block update operations when online backup
is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I also agree that page cache enhancement is interesting, but probably
should be tackled as a separate project. But keeping this goal in mind
while making changes for backup is a good thing. An interface that
that allows backup to use/reuse a single buffer in the page cache seems
reasonable. Specializing it would seem to allow some optimizations
where free page searching could be avoided for this operation which at
a very low level is going to be pushing/pulling pages as fast as possible.
I have seen the following ideas work well in a weight based page cache,
it tries to limit the overhead of weights by using multiple lru, but
still have some of the benefit of weight based scheme:
1) have a much smaller range than 0-100, something like 5 where each
value is it's own lru queue. This reduces the overhead of searching
and sorting based on weight.
2) as dan suggests, something like:
no weight: free list
0: backup page, linear scan heap pages, read ahead,
1: probe accessed heap page
2: leaf page
3: non-leaf page
4: root
3) to account for re-reference, pages move up in value when
re-referenced. Revalue happens only when page is accessed so
page is already latched, so limits additional overhead needed
to reweigh page.
various methods can be used for moving down in value:
o whole queues at a time
o individual pages in lru order, based on some sort of clock like
current clock
Øystein Grøvlen wrote:
>>>>>>"DJD" == Daniel John Debrunner <dj...@debrunners.com> writes:
>
>
> DJD> I think modifications to the cache would be useful for b), so
> DJD> that entries in the cache (through generic apis, not specific
> DJD> to store) could mark how "useful/valuable" they are. Just a
> DJD> simple scheme, lower numbers less valuable, higher numbers
> DJD> more valuable, and if it makes it easier to fix a range,
> DJD> e.g. 0-100, then that would be ok. Then the store could added
> DJD> pages to the cache with this weighting, e.g. (to get the
> DJD> general idea)
>
> DJD> pages for backup - weight 0
> DJD> overflow column pages - weight 10
> DJD> regular pages - weight 20
> DJD> leaf index pages - weight 30
> DJD> root index pages 80
>
> DJD> This weight would then be factored into the decision to throw pages out
> DJD> or not.
>
> I agree that we need some mechanism to prevent operations from filling
> the cache with pages that is not likely to be accesssed again in the
> near future. However, I am afraid that a very detailed "cost-based"
> scheme may create a significant overhead compared to a simple LRU
> scheme.
>
> One may operate with separate LRU queues for different weights, but I
> guess the number of possible weights will have to be restricted in
> that case.
>
> I am also not convinced that it is the type of page that is the most
> important criteria for caching. What matters is access frequency.
> The page type may give a hint, but leaf pages of one index may be more
> frequently accessed than root pages of other indexes.
>
> The type of access is also a relevant criteria. Pages accessed
> sequentially is often less likely to be accessed again in the near
> future than pages accessed by direct lookup. A separate LRU queue for
> sequentially accessed pages may prevent backup and other sequentially
> scans (e.g., select * from t) from throwing out directly accessed
> pages (e.g., index pages and data pages accessed through indexes.)
>
> DJD> This project could be independent of the online backup and could have
> DJD> benfits elsewhere.
>
> I agree.
>
>
Re: page replacement [jira] Commented: (DERBY-239) Need a online
backup feature that does not block update operations when online backup is
in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "DJD" == Daniel John Debrunner <dj...@debrunners.com> writes:
DJD> I think modifications to the cache would be useful for b), so
DJD> that entries in the cache (through generic apis, not specific
DJD> to store) could mark how "useful/valuable" they are. Just a
DJD> simple scheme, lower numbers less valuable, higher numbers
DJD> more valuable, and if it makes it easier to fix a range,
DJD> e.g. 0-100, then that would be ok. Then the store could added
DJD> pages to the cache with this weighting, e.g. (to get the
DJD> general idea)
DJD> pages for backup - weight 0
DJD> overflow column pages - weight 10
DJD> regular pages - weight 20
DJD> leaf index pages - weight 30
DJD> root index pages 80
DJD> This weight would then be factored into the decision to throw pages out
DJD> or not.
I agree that we need some mechanism to prevent operations from filling
the cache with pages that is not likely to be accesssed again in the
near future. However, I am afraid that a very detailed "cost-based"
scheme may create a significant overhead compared to a simple LRU
scheme.
One may operate with separate LRU queues for different weights, but I
guess the number of possible weights will have to be restricted in
that case.
I am also not convinced that it is the type of page that is the most
important criteria for caching. What matters is access frequency.
The page type may give a hint, but leaf pages of one index may be more
frequently accessed than root pages of other indexes.
The type of access is also a relevant criteria. Pages accessed
sequentially is often less likely to be accessed again in the near
future than pages accessed by direct lookup. A separate LRU queue for
sequentially accessed pages may prevent backup and other sequentially
scans (e.g., select * from t) from throwing out directly accessed
pages (e.g., index pages and data pages accessed through indexes.)
DJD> This project could be independent of the online backup and could have
DJD> benfits elsewhere.
I agree.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12316434 ]
>
> Suresh Thalamati commented on DERBY-239:
> ----------------------------------------
[comments just on this issue]
> b) read each pages in to the page cache first and then latch the
> page in the cache until a temporary copy of it is made. This approach
> does not have extra overhead of extra latches on the page keys during writes , but
> will pollute the page cache with the pages that are only required by the
> backup; this might have impact on user operations because active user pages may
> have been replaced by the backup pages in the page cache.
>
> or
>
> c) read pages into buffer pool and latch them while making a copy similar to
> the above approach, but some how make sure that user pages are not kicked out
> of the buffer pool.
I think b) is the correct approach, but what is 'buffer pool' in c)?
I think modifications to the cache would be useful for b), so that
entries in the cache (through generic apis, not specific to store) could
mark how "useful/valuable" they are. Just a simple scheme, lower numbers
less valuable, higher numbers more valuable, and if it makes it easier
to fix a range, e.g. 0-100, then that would be ok. Then the store could
added pages to the cache with this weighting, e.g. (to get the general idea)
pages for backup - weight 0
overflow column pages - weight 10
regular pages - weight 20
leaf index pages - weight 30
root index pages 80
This weight would then be factored into the decision to throw pages out
or not.
This project could be independent of the online backup and could have
benfits elsewhere.
Dan.
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>Another candidate for the Wiki site?
>
>
> The functional spec should at least be attached to the Jira entry.
>
> Dan.
>
>
Good point, will attach it to the jira for the moment. Have not
figured out yet, how Wiki works.
Thanks
-suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> Another candidate for the Wiki site?
The functional spec should at least be attached to the Jira entry.
Dan.
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Another candidate for the Wiki site?
David
Suresh Thalamati wrote:
>
> Attached is the functional/design spec based on the input I got from
> everyone on this topic. I think, most of the issues are covered in this
> doc, except the backup of jars stored inside the database.
>
>
> Thanks
> -suresh
>
>
> ------------------------------------------------------------------------
>
> Online Backup Functional/Design Spec
>
>
>
>
> Overview
>
>
>
> Support real-time */online backup/* mechanism that does not block
> database operations when backup is in progress. When backup is running,
> currently Derby only permits reads from the database. By adding support
> to permit writes, it will be easier for the users to schedule backups
> without being concerned about the impact of backup operation on the
> applications that are accessing the database.
>
>
>
>
> Introduction
>
>
>
> New online backup mechanism will not change the way database can be
> restored from the backup. In the current system, if there any non-logged
> operations when backup starts, they will be rolled back on restore
> because database is frozen at the start of the backup. To support
> real-time online backups that do not block user operations, it is
> necessary to have the transaction log for all the operations that happen
> after the backup starts to bring the database to the consistent state
> during restore. Because a transaction that has non-logged operations
> can commit when backup is in progress. To make a consistent backup of
> the database, there should not be any pending transactions with
> non-logged operations.
>
>
>
>
> JDBC
>
>
>
> Backup operations are support through system procedures. They can be
> called using JDBC program or through IJ.
>
>
> Existing Backup Procedures:
>
> After this enhancement, existing backup procedures
> (SYSCS_UTIL.SYSCS_BACKUP_DATABASE ...etc) will allow writes to the
> database.
>
> Information about existing online backup procedures can be found in the
> Derby admin guide. Derby Admin Guide
> <http://db.apache.org/derby/docs/dev/adminguide/> . These backup
> procedures will *WAIT *for the transaction with non-logged operations to
> commit before proceeding with the backup. __
>
>
> New Procedures:
>
> Two new procedures are added to make backup wait/return error, if any
> non-logged operations are pending at the start of the backup.
>
>
>
> Full backup:
>
> */SYSCS_UTIL.SYSCS_BACKUP_DATABASE(IN BACKUPDIR VARCHAR(), /*
>
> */ /_IN WAIT_FOR_NONLOGGED_OPERATIONS
> SMALLINT_/) /*
>
>
> Full backup with log archive mode:
>
> *SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE
> (IN BACKUPDIR VARCHAR(32672), *
>
> * IN SMALLINT DELETE_ARCHIVED_LOG_FILES,
> /_IN WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT_/_))_
>
> *
> Arguments
>
>
>
> */BACKUPDIR:/**//*
>
> An input argument of type VARCHAR(32672) that specifies the full system
> path to the database directory to be backed up.
>
>
>
> */DELETE_ARCHIVED_LOG_FILES:/*
>
> A non-zero input argument of type SMALLINT will make backup procedures
> to delete existing online archived log files that were created before
> this backup will be deleted. The log files are deleted only after a
> successful backup.
>
> * *
>
> */WAIT_FOR_NONLOGGED_OPERATIONS/* :
> A non-zero input argument of type SMALLINT will make backup procedures
> to wait for the non-logged operation that are currently running to
> commit, while an argument of 0 will make backup fail if there are any
> uncommitted non-logged operations.
>
>
>
>
> Design
>
>
>
> Current Online backup copies all the database files (log, seg0 ...Etc)
> to the specified location when the database is frozen (writes are blocked).
>
> In the new scheme, goal is to copy the database information to the
> backup location, without blocking any user operation for the whole
> duration of the backup. Following sections discuss how the information
> is copied into the backup.
>
>
>
>
> Containers:
>
>
>
> At the start of the backup, all the containers that need to be backed up
> are found by finding all the files in the data segment (seg0). Then each
> container is copied to the backup by reading through the page cache
> instead of the direct file copy.
>
>
>
> � Individual pages are latched, when they are being copied to the
> backup to prevent modifications to the page.
>
> � Any new containers created when backup is in progress will be
> recreated on restore using the information from the transaction log.
>
> � Containers that were in the process of creation will not be
> backed up, if the creation is not complete. Container cache will make
> sure that backup can not get handle to the containers that are being
> created.
>
> � If committed drop occurs on a container that is being backed
> up, backup will wait for the committed drop to complete and then copy
> the stub to the backup.
>
> � Committed container stubs that are created after the checkpoint
> for the backup will not be deleted until the backup is complete. A
> checkpoint that occurs after the backup garbage collects them.
>
> � All requests to open containers in non-logged mode will be
> automatically converted to logged mode after the backup starts.
>
>
>
>
> Log files:
>
> Transaction log is used to bring the database to the consistent state on
> restore from the backup. A checkpoint is made at the start of the
> backup; this checkpoint will be starting point for the recovery from the
> backup. Log files are copied to the backup using regular file copy.
>
>
>
> * Backup Checkpoint information in the log control files is copied
> to the backup location at the start.
> * All the transaction log files starting from the backup checkpoint
> to the log file that is active at the end of the backup are copied
> into the backup.
> * State of the database in the backup depends on last log file that
> goes into the back up. After all the containers are copied to the
> backup , last log file that goes into the backup is determined.
> * Database engine wills not delete the log files that are to yet to
> be backed up, when backup is running even if they are no longer
> required for regular crash recovery.
> * Checkpoints can occur in parallel when the backup is progress.
>
>
>
>
> Jar files:
>
>
>
> Storing Jar files in the database is not a logged operations and they
> don�t go through the page cache; they will be copied to the backup
> directly as it is done now. I am not sure at the moment, if that will
> lead to any issues. Any input about jar files will be helpful.
>
>
>
>
> Other files:
>
> * Database bootstrap file Service.Properties will be to be copied to
> the backup. Need to check if any synchronization is required to
> copy this file.
> * Copy the BACKUP.HISTORY file at the end of the backup.
> * Lock files should not be copied to the backup.
> * Tmp directory should not be backed up.
>
>
>
>
>
>
> Testing:
>
> Most of the current backup tests are single threaded. I will add new
> functional tests case that will execute read/writes on the database,
> when backup is running. Some of the testing scenarios:
>
> 1. Test the new procedure calls.
>
> 2. Checkpoint in parallel to the backup.
>
> 3. DML operations in parallel to the backup.
>
> 4. Grow the container after it is backed up / parallel.
>
> 5. Non-logged operations in parallel to the backup and pending when
> backup starts.
>
> 6. DDL operations in parallel to the backup.
>
> 7. In-Place compress in parallel to the backup.
>
> 8. �.etc.
>
>
>
>
>
>
> References:
>
>
>
> � Derby Admin Guide Backup Section
> <http://db.apache.org/derby/docs/dev/adminguide/>
>
> � Derby-Dev Thread
> <http://nagoya.apache.org/eyebrowse/ReadMsg?listId=270&msgNo=5038>
>
>
>
>
>
>
>
> � Version 1. 10/5/2005 : suresh.thalamati@gmail.com
>
>
>
>
>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Attached is the functional/design spec based on the input I got from
everyone on this topic. I think, most of the issues are covered in
this doc, except the backup of jars stored inside the database.
Thanks
-suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I don't think I followed all the checkpoint issues, but while
thinking about it I had these thoughts:
o there are at least 2 reasons to have checkpoints:
1) allow garbage collection of log file space, if logs are
not required for rollfoward backup recovery.
2) to minimize the amount of redo work necessary if
a crash occurs.
With respect to option #2 I think you have have to be careful in
online backup, I think in the backup you need to set the redo
low water mark to a checkpoint that predates the start of the
backup, you can not use a subsequent checkpoint that happens
during the backup. This is because you are just going to copy
data pages in order and the backup copy may have a page version
before those represented in a checkpoint.
Suresh Thalamati wrote:
> Mike Matrigali wrote:
>
>> o I think it would seem reasonable to leave decision of whether to wait
>> for unlogged operations to finish up to the caller of the backup.
>> For some set of users waiting is fine, they just want the backup
>> eventually, and they are using the feature because they want normal
>> users to be impacted as little as possible (at least with respect to
>> lock concurrency). Others may need to know imediately why something
>> is blocking their operation.
>>
>
> That is a good point, but to provide both the options; new backup
> procedures that takes an extra parameter for the users to specify
> wait/fail are required.
>
> I am thinking of providing following two additional backup procedures
> for this case:
>
> 1) Full backup
> SYSCS_UTIL.SYSCS_BACKUP_DATABASE(IN BACKUPDIR VARCHAR(),
> IN WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT)
>
> and
>
> 2) Full backup with log archive mode:
>
> SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE
> (IN BACKUPDIR VARCHAR(32672), IN SMALLINT DELETE_ARCHIVED_LOG_FILES,
> IN WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT))
>
>
> WAIT_FOR_NONLOGGED_OPERATIONS :
> A non-zero input argument of type SMALLINT will make backup procedures
> to wait for the non-logged operation that are currently running to
> commit, while an argument of 0 will make backup fail if there are any
> uncommitted non-logged operations.
>
> Please note that the signature of the these procedures is same as
> the current procedures , except for the additional argument.
>
> Current procedures has to be maintained for backward compatibility
> reasons. They will work in non-blocking mode(I doubt any user will
> complain about it!). Existing backup procedures will wait for non-logged
> operations commit.
>
> Any comments ?
>
>
>> o I had just assumed that online backup would only work in roll foward
>> mode, so there was really no garbage collection anyway - at least not
>> until we provide a way to move rollforward logs somewhere
>> automatically. I think this is sort of along the lines of what Oystein
>> is saying, would be good to do in future but 1st impl does not need to
>> address. Remember while Derby does not have built in size limits, it
>> does not have features one expects from VLDB (for really large db's you
>> probably want to have multiple threads/devices backup up in parallel
>> along with simultaneous log file moves).
>>
>
> Mike, I think it will be more helpful to the users if Derby supports
> non-blocking simple full backups, not just non-blocking backups only
> with log archive mode enabled.
>
> I believe Øystein is referring to the non lograchive mode scenario, if
> there are checkpoints when backup is in progress; log space can released
> by copying the log files to the backup in parallel to the data files
> copy operation. I also think, this enhancement is not essential for
> first implementation of real time online backup.
>
>
> Thanks
> -suresht
>
>
>
>>
>>
>> Øystein Grøvlen wrote:
>>
>>
>>>>>>>> "ØG" == Øystein Grøvlen <Oy...@Sun.COM> writes:
>>>
>>>
>>>
>>>>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
>>>
>>>
>>> ST> My initial thought is to simply disable
>>> garbage-collection of log
>>> ST> files for the duration of the backup. unless there are some
>>> specific
>>> ST> advantages in writing backup-start log record.
>>>
>>> ØG> Disabling garabage-collection directly is probably the
>>> cleanest way to
>>> ØG> do this.
>>>
>>> On second thoughts, for large databases, the time to copy the data
>>> files may be so long that the log volume becomes prohibitively large
>>> if garbage collection is turned off for the entire backup. In that
>>> case, it would be better to copy finished log files in parallel with
>>> the data files and let them be garbage collected at any time after it
>>> has been copied. This probably does not need to be part of a first
>>> implementation, but something one should consider for the future.
>>>
>>
>>
>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Mike Matrigali wrote:
> o I think it would seem reasonable to leave decision of whether to wait
> for unlogged operations to finish up to the caller of the backup.
> For some set of users waiting is fine, they just want the backup
> eventually, and they are using the feature because they want normal
> users to be impacted as little as possible (at least with respect to
> lock concurrency). Others may need to know imediately why something
> is blocking their operation.
>
That is a good point, but to provide both the options; new backup
procedures that takes an extra parameter for the users to specify
wait/fail are required.
I am thinking of providing following two additional backup procedures
for this case:
1) Full backup
SYSCS_UTIL.SYSCS_BACKUP_DATABASE(IN BACKUPDIR VARCHAR(),
IN WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT)
and
2) Full backup with log archive mode:
SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE
(IN BACKUPDIR VARCHAR(32672), IN SMALLINT DELETE_ARCHIVED_LOG_FILES,
IN WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT))
WAIT_FOR_NONLOGGED_OPERATIONS :
A non-zero input argument of type SMALLINT will make backup procedures
to wait for the non-logged operation that are currently running to
commit, while an argument of 0 will make backup fail if there are any
uncommitted non-logged operations.
Please note that the signature of the these procedures is same as
the current procedures , except for the additional argument.
Current procedures has to be maintained for backward compatibility
reasons. They will work in non-blocking mode(I doubt any user will
complain about it!). Existing backup procedures will wait for
non-logged operations commit.
Any comments ?
> o I had just assumed that online backup would only work in roll foward
> mode, so there was really no garbage collection anyway - at least not
> until we provide a way to move rollforward logs somewhere
> automatically. I think this is sort of along the lines of what Oystein
> is saying, would be good to do in future but 1st impl does not need to
> address. Remember while Derby does not have built in size limits, it
> does not have features one expects from VLDB (for really large db's you
> probably want to have multiple threads/devices backup up in parallel
> along with simultaneous log file moves).
>
Mike, I think it will be more helpful to the users if Derby supports
non-blocking simple full backups, not just non-blocking backups only
with log archive mode enabled.
I believe Øystein is referring to the non lograchive mode scenario, if
there are checkpoints when backup is in progress; log space can
released by copying the log files to the backup in parallel to the
data files copy operation. I also think, this enhancement is not
essential for first implementation of real time online backup.
Thanks
-suresht
>
>
> Øystein Grøvlen wrote:
>
>
>>>>>>>"ØG" == Øystein Grøvlen <Oy...@Sun.COM> writes:
>>
>>
>>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>>
>> ST> My initial thought is to simply disable garbage-collection of log
>> ST> files for the duration of the backup. unless there are some specific
>> ST> advantages in writing backup-start log record.
>>
>> ØG> Disabling garabage-collection directly is probably the cleanest way to
>> ØG> do this.
>>
>>On second thoughts, for large databases, the time to copy the data
>>files may be so long that the log volume becomes prohibitively large
>>if garbage collection is turned off for the entire backup. In that
>>case, it would be better to copy finished log files in parallel with
>>the data files and let them be garbage collected at any time after it
>>has been copied. This probably does not need to be part of a first
>>implementation, but something one should consider for the future.
>>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
o I think it would seem reasonable to leave decision of whether to wait
for unlogged operations to finish up to the caller of the backup.
For some set of users waiting is fine, they just want the backup
eventually, and they are using the feature because they want normal
users to be impacted as little as possible (at least with respect to
lock concurrency). Others may need to know imediately why something
is blocking their operation.
o I had just assumed that online backup would only work in roll foward
mode, so there was really no garbage collection anyway - at least not
until we provide a way to move rollforward logs somewhere
automatically. I think this is sort of along the lines of what Oystein
is saying, would be good to do in future but 1st impl does not need to
address. Remember while Derby does not have built in size limits, it
does not have features one expects from VLDB (for really large db's you
probably want to have multiple threads/devices backup up in parallel
along with simultaneous log file moves).
Øystein Grøvlen wrote:
>>>>>>"ØG" == Øystein Grøvlen <Oy...@Sun.COM> writes:
>
>
>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>
> ST> My initial thought is to simply disable garbage-collection of log
> ST> files for the duration of the backup. unless there are some specific
> ST> advantages in writing backup-start log record.
>
> ØG> Disabling garabage-collection directly is probably the cleanest way to
> ØG> do this.
>
> On second thoughts, for large databases, the time to copy the data
> files may be so long that the log volume becomes prohibitively large
> if garbage collection is turned off for the entire backup. In that
> case, it would be better to copy finished log files in parallel with
> the data files and let them be garbage collected at any time after it
> has been copied. This probably does not need to be part of a first
> implementation, but something one should consider for the future.
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ØG" == Øystein Grøvlen <Oy...@Sun.COM> writes:
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> My initial thought is to simply disable garbage-collection of log
ST> files for the duration of the backup. unless there are some specific
ST> advantages in writing backup-start log record.
ØG> Disabling garabage-collection directly is probably the cleanest way to
ØG> do this.
On second thoughts, for large databases, the time to copy the data
files may be so long that the log volume becomes prohibitively large
if garbage collection is turned off for the entire backup. In that
case, it would be better to copy finished log files in parallel with
the data files and let them be garbage collected at any time after it
has been copied. This probably does not need to be part of a first
implementation, but something one should consider for the future.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> Suresh Thalamati wrote:
>> I think one of the following options may be more acceptable:
>> 1) make backup/archive mode process wait until
>> all the transaction that has the non-logged
>> operations are committed.
>> or
>>
>> 2) convert the non-logged operation to logging mode after
>> flushing the containers , once the backup starts.
>>
>> My preference is option 1) , it might be less complicated than
>> option 2).
>>
ST> On my way back home , I was thinking may be better option is :
ST> 3) to make backup/(log archive mode enabling) fail , if there are
ST> uncommitted transactions with non-logged operations ? instead of
ST> making backup process wait for the non-logged operations to
ST> commit.
I think your last suggestion is a good idea. If, for example, an
import takes very long, it might be better to flag the conflict
immediately instead of executing the backup several hours later.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Suresh Thalamati wrote:
> Øystein Grøvlen wrote:
>
>>>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
>>>>>>>
>>>>>>
>>
>> .....<snip>
>> ST> I agree it by writing log record for start of backup, we can
>> prevent
>> ST> garbage-collection of log files.
>>
>> ST> My initial thought is to simply disable
>> garbage-collection of log
>> ST> files for the duration of the backup. unless there are some
>> specific
>> ST> advantages in writing backup-start log record.
>>
>> Disabling garabage-collection directly is probably the cleanest way to
>> do this.
>>
>> How will you determine where to start the redo scan at recovery? Do
>> you need some mark in the log for that purpose?
>>
>>
>>
> I believe using the checkpoint information available at the start of
> the backup, redo scan staring point can be determined at recovery.
> log.ctrl file contains the checkpoint information, this file
> should be copied to the backup after disabling the log-file garbage
> collection,
> but before stating data files copy operation.
>
>
>>
>>
>> >> Generally, we cannot give a guarantee that operations that are
>> >> performed during backup are reflected in the backup. If I have
>> >> understand correctly, transactions that commits after the data
>> copying
>> >> is finished, will not be reflected. Since a user will not be
>> able to
>> >> distiguish between operations committed during data copying and
>> >> operations committed during log copying, he cannot be sure
>> concurrent
>> >> operations is reflected in the backup.
>> >> >>
>>
>> ST> I agree with you that , one can not absolutely guarantee that
>> ST> backup will include operations committed till a particular
>> ST> time are included in the backup. But the backup design
>> ST> depends on the transactions log to bring the database to
>> ST> consistent state , because when data files are being copied ,
>> ST> it is possible that some of the page are written to the disk.
>> ST> So we need the transaction log until the data files are copied
>> ST> for sure. If a user commits a non-logged operation when data
>> ST> files are being copied , he/she would expect it to be there in
>> ST> the backup, similar to a logged operation.
>>
>> My point was that a user will not be able to distiguish between the
>> data file copying period and the log copying period. Hence, he does
>> not know whether his operation was committed while the data files was
>> being copied.
>>
>> ST> Please note that non-logging operation in Derby are not
>> ST> explicit to the users, most of non-logging stuff is done by
>> ST> the system without the user knowledge.
>>
>> I understand.
>>
>>
>> >> This is not more of an issue for a new backup mechanism than it is
>> >> currently for roll-forward recovery. Roll-forward recovery
>> will not be
>> >> able recover non-logged operations either.
>>
>> ST> Yes. Roll-forward recovery has same issues, once the log
>> ST> archive mode that is required for roll-forward recovery is
>> ST> enabled all the operations are logged including the operations
>> ST> that are not logged normally like create index. But I think
>> ST> the currently derby does not handle correctly . it does not
>> ST> force logging for non-logged operations that were started
>> ST> before log archive mode is enabled.
>>
>> The cheapest way to handle non-logged operations that started before
>> backup/archive mode enabling, is to just make them fail and roll them
>> back. I think that would be an acceptable solution.
>>
>>
>>
>
> I like the idea, but I am not sure how users will react if an
> operation fails in the middle because of a backup/archive mode
> I think one of the following options may be more acceptable:
> 1) make backup/archive mode process wait until
> all the transaction that has the non-logged
> operations are committed.
> or
>
> 2) convert the non-logged operation to logging mode after
> flushing the containers , once the backup starts.
>
> My preference is option 1) , it might be less complicated than option
> 2).
>
On my way back home , I was thinking may be better option is :
3) to make backup/(log archive mode enabling) fail , if there
are uncommitted transactions with non-logged operations ? instead of
making backup process wait for the non-logged operations to
commit.
>
>> >> If users needs that, we
>> >> should provide logged version of these operations.
>> >> >>
>> ST> I think, during backup non-logged operations should be
>> logged by the
>> ST> system or block them.
>> I think blocking them should be acceptable to most users.
>>
>>
>>
>
> I think converting non-logged operations to logged operations may be
> a better choice. If a user wants to create indexes/import small
> amount of data during backup , they will still be able to do.
> In case if user is concerned about performance of these operation ,
> they can stop the
> backup or wait until backup is done. If the database is in the log
> archive mode ,
> they can disable using SYSC_UTIL.SYSC_DISABLE_LOG_ARCHIVE_MODE.
> and re enable the archive mode with a fresh full backup.
>
>
>> ST> If user is really concerned of performance they will not
>> ST> execute them in parallel.
>>
>> This advice may work for backup, but not for enabling roll-forward
>> recovery. If I was user that was concerned with performance, I think
>> I would prefer to still create an index unlogged and rather recreate
>> it if recovery is needed. (I guess this would require roll-forward
>> recovery to ignore updates to non-existing indexes.) I could limit
>> the vulnerability by making a backup after unlogged operations have
>> been performed.
>>
>>
>>
> I like the idea of rebuilding the indexes during recovery , but we
> may want to do it as
> a different project.
>
>> By the way, how is normal recovery of unlogged operations handled? Is
>> the commit of unlogged operations delayed until all data pages created
>> by the operation have been flushed to disk?
>>
>>
>>
>
> Yes. I think at the commit time all unlogged containers pages in
> the cache are flushed to the disk.
> To my knowledge, all the non logged operation happen on new
> containers and the container creation part is logged ,
> if a crash occurs before the commit . container will be dropped by
> the rollback of the CREATE log record.
>
>
> Thanks
> -suresh
>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen wrote:
>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>>>>>>
>>>>>>
>
>.....<snip>
>
> ST> I agree it by writing log record for start of backup, we can prevent
> ST> garbage-collection of log files.
>
> ST> My initial thought is to simply disable garbage-collection of log
> ST> files for the duration of the backup. unless there are some specific
> ST> advantages in writing backup-start log record.
>
>Disabling garabage-collection directly is probably the cleanest way to
>do this.
>
>How will you determine where to start the redo scan at recovery? Do
>you need some mark in the log for that purpose?
>
>
>
I believe using the checkpoint information available at the start of
the backup, redo scan staring point can be determined at recovery.
log.ctrl file contains the checkpoint information, this file
should be copied to the backup after disabling the log-file garbage
collection,
but before stating data files copy operation.
>
>
>
> >> Generally, we cannot give a guarantee that operations that are
> >> performed during backup are reflected in the backup. If I have
> >> understand correctly, transactions that commits after the data copying
> >> is finished, will not be reflected. Since a user will not be able to
> >> distiguish between operations committed during data copying and
> >> operations committed during log copying, he cannot be sure concurrent
> >> operations is reflected in the backup.
> >>
> >>
>
>
> ST> I agree with you that , one can not absolutely guarantee that
> ST> backup will include operations committed till a particular
> ST> time are included in the backup. But the backup design
> ST> depends on the transactions log to bring the database to
> ST> consistent state , because when data files are being copied ,
> ST> it is possible that some of the page are written to the disk.
> ST> So we need the transaction log until the data files are copied
> ST> for sure. If a user commits a non-logged operation when data
> ST> files are being copied , he/she would expect it to be there in
> ST> the backup, similar to a logged operation.
>
>My point was that a user will not be able to distiguish between the
>data file copying period and the log copying period. Hence, he does
>not know whether his operation was committed while the data files was
>being copied.
>
> ST> Please note that non-logging operation in Derby are not
> ST> explicit to the users, most of non-logging stuff is done by
> ST> the system without the user knowledge.
>
>I understand.
>
>
> >> This is not more of an issue for a new backup mechanism than it is
> >> currently for roll-forward recovery. Roll-forward recovery will not be
> >> able recover non-logged operations either.
>
> ST> Yes. Roll-forward recovery has same issues, once the log
> ST> archive mode that is required for roll-forward recovery is
> ST> enabled all the operations are logged including the operations
> ST> that are not logged normally like create index. But I think
> ST> the currently derby does not handle correctly . it does not
> ST> force logging for non-logged operations that were started
> ST> before log archive mode is enabled.
>
>The cheapest way to handle non-logged operations that started before
>backup/archive mode enabling, is to just make them fail and roll them
>back. I think that would be an acceptable solution.
>
>
>
I like the idea, but I am not sure how users will react if an
operation fails in the middle because of a backup/archive mode
I think one of the following options may be more acceptable:
1) make backup/archive mode process wait until all the
transaction that has the non-logged
operations are committed.
or
2) convert the non-logged operation to logging mode after
flushing the containers , once the backup starts.
My preference is option 1) , it might be less complicated than option 2).
> >> If users needs that, we
> >> should provide logged version of these operations.
> >>
> >>
>
> ST> I think, during backup non-logged operations should be logged by the
> ST> system or block them.
>
>I think blocking them should be acceptable to most users.
>
>
>
I think converting non-logged operations to logged operations may be a
better choice.
If a user wants to create indexes/import small amount of data during
backup , they will still be able to do.
In case if user is concerned about performance of these operation , they
can stop the
backup or wait until backup is done. If the database is in the log
archive mode ,
they can disable using SYSC_UTIL.SYSC_DISABLE_LOG_ARCHIVE_MODE.
and re enable the archive mode with a fresh full backup.
> ST> If user is really concerned of performance they will not
> ST> execute them in parallel.
>
>This advice may work for backup, but not for enabling roll-forward
>recovery. If I was user that was concerned with performance, I think
>I would prefer to still create an index unlogged and rather recreate
>it if recovery is needed. (I guess this would require roll-forward
>recovery to ignore updates to non-existing indexes.) I could limit
>the vulnerability by making a backup after unlogged operations have
>been performed.
>
>
>
I like the idea of rebuilding the indexes during recovery , but we
may want to do it as
a different project.
>By the way, how is normal recovery of unlogged operations handled? Is
>the commit of unlogged operations delayed until all data pages created
>by the operation have been flushed to disk?
>
>
>
Yes. I think at the commit time all unlogged containers pages in the
cache are flushed to the disk.
To my knowledge, all the non logged operation happen on new
containers and the container creation part is logged ,
if a crash occurs before the commit . container will be dropped by
the rollback of the CREATE log record.
Thanks
-suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> Thanks for the input , Øystein. My comments are in-line. Øystein
ST> Grøvlen wrote:
>> I guess the idea is to do a log switch when the data files have been
>> copied and copy all log from before this log switch.
ST> That's what I plan to do, except the the log switch will not
ST> happen until it is time to determine the last log file that
ST> needs to go into the backup after copying the data files and
ST> the rest of the log files. From the user perspective ,
Good idea. Then the uncertainty of what really has made it into the
backup is limited to just a few seconds before it completed.
ST> Derby online backup will include all the transactions that
ST> were committed the backup. i.e if the backup starts at 9PM
ST> and ends at 10PM, backup will have data approximately until
ST> 10PM almost , not 9PM.
ST> My understanding of what goes into the backup is that , data
ST> in the backup need not match to the exact real-time. please
ST> correct if this is wrong assumption.
I agree with you.
>> One will have to
>> prevent any log files that is needed from being garbage collected
>> before they are copied. I guess that can be achieved by running
>> backup in a transaction and log the start of backup. That way, the
>> open backup transaction should prevent garbage-collection of relevant
>> log files.
>>
>>
ST> I agree it by writing log record for start of backup, we can prevent
ST> garbage-collection of log files.
ST> My initial thought is to simply disable garbage-collection of log
ST> files for the duration of the backup. unless there are some specific
ST> advantages in writing backup-start log record.
Disabling garabage-collection directly is probably the cleanest way to
do this.
How will you determine where to start the redo scan at recovery? Do
you need some mark in the log for that purpose?
>> One question I have is whether compensation log records (CLR) in Derby
>> are self-contained. If they depend on the original non-CLR, log
>> records for transactions that were rolled back and terminated long
>> before the backup is finished, will be needed to be able to redo the
>> CLR.
>>
ST> I think CLR's are not self contained in Derby. CLR log
ST> records contain, log instant of the original operation log
ST> record that was rolled back.
If garbage-collection is disabled this does probably not matter, but
this means that a start-of-backup log record is not sufficient to
prevent garbage-collection of relevant log records.
>> Generally, we cannot give a guarantee that operations that are
>> performed during backup are reflected in the backup. If I have
>> understand correctly, transactions that commits after the data copying
>> is finished, will not be reflected. Since a user will not be able to
>> distiguish between operations committed during data copying and
>> operations committed during log copying, he cannot be sure concurrent
>> operations is reflected in the backup.
>>
>>
ST> I agree with you that , one can not absolutely guarantee that
ST> backup will include operations committed till a particular
ST> time are included in the backup. But the backup design
ST> depends on the transactions log to bring the database to
ST> consistent state , because when data files are being copied ,
ST> it is possible that some of the page are written to the disk.
ST> So we need the transaction log until the data files are copied
ST> for sure. If a user commits a non-logged operation when data
ST> files are being copied , he/she would expect it to be there in
ST> the backup, similar to a logged operation.
My point was that a user will not be able to distiguish between the
data file copying period and the log copying period. Hence, he does
not know whether his operation was committed while the data files was
being copied.
ST> Please note that non-logging operation in Derby are not
ST> explicit to the users, most of non-logging stuff is done by
ST> the system without the user knowledge.
I understand.
>> This is not more of an issue for a new backup mechanism than it is
>> currently for roll-forward recovery. Roll-forward recovery will not be
>> able recover non-logged operations either.
ST> Yes. Roll-forward recovery has same issues, once the log
ST> archive mode that is required for roll-forward recovery is
ST> enabled all the operations are logged including the operations
ST> that are not logged normally like create index. But I think
ST> the currently derby does not handle correctly . it does not
ST> force logging for non-logged operations that were started
ST> before log archive mode is enabled.
The cheapest way to handle non-logged operations that started before
backup/archive mode enabling, is to just make them fail and roll them
back. I think that would be an acceptable solution.
>> If users needs that, we
>> should provide logged version of these operations.
>>
>>
ST> I think, during backup non-logged operations should be logged by the
ST> system or block them.
I think blocking them should be acceptable to most users.
ST> If user is really concerned of performance they will not
ST> execute them in parallel.
This advice may work for backup, but not for enabling roll-forward
recovery. If I was user that was concerned with performance, I think
I would prefer to still create an index unlogged and rather recreate
it if recovery is needed. (I guess this would require roll-forward
recovery to ignore updates to non-existing indexes.) I could limit
the vulnerability by making a backup after unlogged operations have
been performed.
By the way, how is normal recovery of unlogged operations handled? Is
the commit of unlogged operations delayed until all data pages created
by the operation have been flushed to disk?
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Thanks for the input , Øystein. My comments are in-line.
Øystein Grøvlen wrote:
>Øystein
>
>
>
>>>>>>"ST(" == Suresh Thalamati (JIRA) <de...@db.apache.org> writes:
>>>>>>
>>>>>>
>
> ST(> Currently backup contains mainly data files (seg0/*) and the
> ST(> transaction log files(log/*) that are there when the backup
> ST(> started. On restore from the backup, transactions are
> ST(> replayed, similar to crash-recovery to bring the database to
> ST(> a consistent state. New online backup also will work same
> ST(> way, except that all the transaction log must be copied to
> ST(> the backup, only after all the data files are backed up.
>
>I guess the idea is to do a log switch when the data files have been
>copied and copy all log from before this log switch.
>
That's what I plan to do, except the the log switch will not happen
until it is time to determine the last log
file that needs to go into the backup after copying the data files and
the rest of the log files. From the user perspective ,
Derby online backup will include all the transactions that were
committed the backup. i.e if the backup starts at
9PM and ends at 10PM, backup will have data approximately until 10PM
almost , not 9PM.
My understanding of what goes into the backup is that , data in the
backup need not match to the
exact real-time. please correct if this is wrong assumption.
>One will have to
>prevent any log files that is needed from being garbage collected
>before they are copied. I guess that can be achieved by running
>backup in a transaction and log the start of backup. That way, the
>open backup transaction should prevent garbage-collection of relevant
>log files.
>
>
>
I agree it by writing log record for start of backup, we can prevent
garbage-collection of log files.
My initial thought is to simply disable garbage-collection of log
files for the duration of the backup.
unless there are some specific advantages in writing backup-start log
record.
>One question I have is whether compensation log records (CLR) in Derby
>are self-contained. If they depend on the original non-CLR, log
>records for transactions that were rolled back and terminated long
>before the backup is finished, will be needed to be able to redo the
>CLR.
>
I think CLR's are not self contained in Derby. CLR log records contain,
log instant of the original
operation log record that was rolled back.
>
>
> ST(> To avoid backup process reading partial written pages, some
> ST(> kind of synchronization mechanism that does not allow reading
> ST(> a page to write to the back backup when the same page is
> ST(> being written to the disk. This can be implemented by one of
> ST(> the following approaches:
>
> ST(> a) By latching on a page key (container id, page number)
> ST(> while doing the write of the page from cache to disk and
> ST(> while reading the page from the disk/cache to write to the
> ST(> backup. This approach has small overhead of acquiring an
> ST(> extra latch during the page cache writes when the backup
> ST(> is in progress.
>
>If I understand correctly, you are here talking about not reading
>pages into the the page cache but copying them to the backup some
>other ways. I do not think that sounds like a good idea since you
>will have to care about the pages that are already in the cache
>anyway.
>
>
>
I agree with you. I was thinking along these lines to use
jdk1.4.2 file to file transfer mechanisms
to have a super-fast backup :-)
>However, I do not understand the part about an extra latch. Don't you
>have to latch a page while it is written to disk anyhow to prevent
>concurrent updates during the write?
>
>
Yes. page is in latched state when it written to the disk. But this
latching mechanism is based
on the page objects in the page cache; Unless backup also goes
through the page cache, it can not
latch the same page object.
> ST(> 2) Committed Non logged operation:
>
> ST(> Basic requirement to have consistent database backup is
> ST(> after the checkpoint for the backup all changes to the
> ST(> database will be available in the transaction log. But
> ST(> Derby provides some non logged operations for performance
> ST(> reasons , for example CREATE INDEX , IMPORT to a empty
> ST(> table ..etc.
>
> ST(> This was not a issue in the old backup mechanism because no
> ST(> operations will be committed once the backup starts. So any
> ST(> non logged operations will be rolled back similar to the
> ST(> regular crash recovery.
>
>Generally, we cannot give a guarantee that operations that are
>performed during backup are reflected in the backup. If I have
>understand correctly, transactions that commits after the data copying
>is finished, will not be reflected. Since a user will not be able to
>distiguish between operations committed during data copying and
>operations committed during log copying, he cannot be sure concurrent
>operations is reflected in the backup.
>
>
>
I agree with you that , one can not absolutely guarantee that
backup will include operations committed till
a particular time are included in the backup. But the backup design
depends on the
transactions log to bring the database to consistent state , because
when data files are being
copied , it is possible that some of the page are written to the disk.
So we need the
transaction log until the data files are copied for sure. If a user
commits a non-logged
operation when data files are being copied , he/she would expect it to
be there in the backup,
similar to a logged operation. Please note that non-logging operation
in Derby are not
explicit to the users, most of non-logging stuff is done by the
system without the user knowledge.
>This is not more of an issue for a new backup mechanism than it is
>currently for roll-forward recovery. Roll-forward recovery will not be
>able recover non-logged operations either.
>
Yes. Roll-forward recovery has same issues, once the log archive mode
that is required for roll-forward recovery
is enabled all the operations are logged including the operations that
are not logged normally like create index.
But I think the currently derby does not handle correctly . it does
not force logging for
non-logged operations that were started before log archive mode is enabled.
> If users needs that, we
>should provide logged version of these operations.
>
>
>
I think, during backup non-logged operations should be logged by the
system or block them.
If user is really concerned of performance they will not execute them
in parallel.
> ST(> 6) checkpoints when the backup is in progress.
>
> ST(> I think it not necessary to allow checkpoints when the backup is in
> ST(> progress. But if some one thinks otherwise , following should
> ST(> be addressed:
>
>Is checkpoints the only way to write data pages to disk? If yes,
>backup cannot block checkpoints since then the backup will no longer
>be non-blocking when the entire page cache becomes dirty.
>
>
>
No. pages are written to the disk by the cache manager to prevent
cache full state.
I believe checkpoint can run in parallel to backup , if I make a
copy of log control file at the
start of the backup and disable log-files garbage collection. On
second thought, I don't
see any reason for not allowing checkpoints during backup.
Thanks
-suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Suresh,
I have some questions/comments. Some of them probably reflects that I
have not yet a detailed understanding of all Store mechanisms.
However, I have earlier participated in the implementation of
non-blocking backup in another RDBMS so I have some knowledge of the
area.
--
Øystein
>>>>> "ST(" == Suresh Thalamati (JIRA) <de...@db.apache.org> writes:
ST(> Currently backup contains mainly data files (seg0/*) and the
ST(> transaction log files(log/*) that are there when the backup
ST(> started. On restore from the backup, transactions are
ST(> replayed, similar to crash-recovery to bring the database to
ST(> a consistent state. New online backup also will work same
ST(> way, except that all the transaction log must be copied to
ST(> the backup, only after all the data files are backed up.
I guess the idea is to do a log switch when the data files have been
copied and copy all log from before this log switch. One will have to
prevent any log files that is needed from being garbage collected
before they are copied. I guess that can be achieved by running
backup in a transaction and log the start of backup. That way, the
open backup transaction should prevent garbage-collection of relevant
log files.
One question I have is whether compensation log records (CLR) in Derby
are self-contained. If they depend on the original non-CLR, log
records for transactions that were rolled back and terminated long
before the backup is finished, will be needed to be able to redo the
CLR.
ST(> To avoid backup process reading partial written pages, some
ST(> kind of synchronization mechanism that does not allow reading
ST(> a page to write to the back backup when the same page is
ST(> being written to the disk. This can be implemented by one of
ST(> the following approaches:
ST(> a) By latching on a page key (container id, page number)
ST(> while doing the write of the page from cache to disk and
ST(> while reading the page from the disk/cache to write to the
ST(> backup. This approach has small overhead of acquiring an
ST(> extra latch during the page cache writes when the backup
ST(> is in progress.
If I understand correctly, you are here talking about not reading
pages into the the page cache but copying them to the backup some
other ways. I do not think that sounds like a good idea since you
will have to care about the pages that are already in the cache
anyway.
However, I do not understand the part about an extra latch. Don't you
have to latch a page while it is written to disk anyhow to prevent
concurrent updates during the write?
ST(> or
ST(> b) read each pages in to the page cache first and then latch
ST(> the page in the cache until a temporary copy of it is
ST(> made. This approach does not have extra overhead of extra
ST(> latches on the page keys during writes , but will pollute
ST(> the page cache with the pages that are only required by
ST(> the backup; this might have impact on user operations
ST(> because active user pages may have been replaced by the
ST(> backup pages in the page cache.
ST(> or
ST(> c) read pages into buffer pool and latch them while making a
ST(> copy similar to the above approach, but some how make sure
ST(> that user pages are not kicked out of the buffer pool.
I think that page cache contention is a separate issue. This is not
just relevant to backup, but to any sequential table scans (e.g.,
select *).
ST(> 2) Committed Non logged operation:
ST(> Basic requirement to have consistent database backup is
ST(> after the checkpoint for the backup all changes to the
ST(> database will be available in the transaction log. But
ST(> Derby provides some non logged operations for performance
ST(> reasons , for example CREATE INDEX , IMPORT to a empty
ST(> table ..etc.
ST(> This was not a issue in the old backup mechanism because no
ST(> operations will be committed once the backup starts. So any
ST(> non logged operations will be rolled back similar to the
ST(> regular crash recovery.
Generally, we cannot give a guarantee that operations that are
performed during backup are reflected in the backup. If I have
understand correctly, transactions that commits after the data copying
is finished, will not be reflected. Since a user will not be able to
distiguish between operations committed during data copying and
operations committed during log copying, he cannot be sure concurrent
operations is reflected in the backup.
This is not more of an issue for a new backup mechanism than it is
currently for roll-forward recovery. Roll-forward recovery will not be
able recover non-logged operations either. If users needs that, we
should provide logged version of these operations.
ST(> 6) checkpoints when the backup is in progress.
ST(> I think it not necessary to allow checkpoints when the backup is in
ST(> progress. But if some one thinks otherwise , following should
ST(> be addressed:
Is checkpoints the only way to write data pages to disk? If yes,
backup cannot block checkpoints since then the backup will no longer
be non-blocking when the entire page cache becomes dirty.
--
Øystein
[jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12316434 ]
Suresh Thalamati commented on DERBY-239:
----------------------------------------
I think providing an online backup mechanism that does not block changes
to the database when the backup is in progress will be a useful feature to Derby users,
especially in the client/server environment. This backup mechanism might take more time
than current online backup because of the synchronization overheads required to allow changes to
the database when backup is in progress. At this point I am not sure how much more
time it will take, but I think it should not be more than 50%, in the worst case scenario.
Current online backup mechanism (that blocks changes to the database) is
supported using system procedures(ex:SYSCS_UTIL.SYSCS_BACKUP_DATABASE ). My
plan is to make the existing backup procedures work work without blocking the
changes to the database; No new system procedures are required. If community thinks
both blocking/non-blocking type backups are useful, new procedures can
be added.
Currently backup contains mainly data files (seg0/*) and the transaction log
files(log/*) that are there when the backup started. On restore from the
backup, transactions are replayed, similar to crash-recovery to bring the database
to a consistent state. New online backup also will work same way, except
that all the transaction log must be copied to the backup, only after all the data
files are backed up.
I think current implementation freezes(no changes to the database) the database
during backup for following reasons :
1) Data files will in a stable state; backup will not contain partially updates
pages on the disk.
2) No new data files will be added/deleted on the disk;
because create/drop operations are blocked.
3) No transaction will committed after the backup starts. So all
unlogged operations will be rolled back.
If the database is not frozen above conditions will not be true, that might
lead to the backups that are in corrupted/inconsistent state. I think, it is
not necessary to freeze the whole database to make a stable backup copy, by
blocking operations that modifies the files on-disk for small amounts of time,
a stable backup can be made.
Following sections explain some of the issues and possible ways to address them to
provide a real online backup that does not block changes to the database for the whole
duration of the backup.
1) Corrupt pages in the backup database:
Backup reads and the page cache writes can be interleaved if the database is
not frozen. i.e it is possible to land up with a page in the backup that has
a portion of the page that is more up-to-date than the rest of the page, if the
page cache writes are not blocked when a page is being read for the backup.
To avoid backup process reading partial written pages, some kind of
synchronization mechanism that does not allow reading a page to write to the
back backup when the same page is being written to the disk. This can be
implemented by one of the following approaches:
a) By latching on a page key (container id, page number) while doing the write
of the page from cache to disk and while reading the page from the
disk/cache to write to the backup. This approach has small overhead of
acquiring an extra latch during the page cache writes when the backup is in progress.
or
b) read each pages in to the page cache first and then latch the
page in the cache until a temporary copy of it is made. This approach
does not have extra overhead of extra latches on the page keys during writes , but
will pollute the page cache with the pages that are only required by the
backup; this might have impact on user operations because active user pages may
have been replaced by the backup pages in the page cache.
or
c) read pages into buffer pool and latch them while making a copy similar to
the above approach, but some how make sure that user pages are not kicked out
of the buffer pool.
One optimization that may be made is to copy the file on the disk as it
is to the backup, but keep track of pages that gets modified when file was
being copied and rewrite those pages by using one of the above latching
mechanisms.
2) Committed Non logged operation:
Basic requirement to have consistent database backup is after the checkpoint
for the backup all changes to the database will be available in the
transaction log. But Derby provides some non logged operations for
performance reasons , for example CREATE INDEX , IMPORT to a empty table
..etc.
This was not a issue in the old backup mechanism because no operations will
be committed once the backup starts. So any non logged operations will be rolled
back similar to the regular crash recovery.
I can think of two ways to address this issue:
a) To block non-logged operations when backup is in progress and also make backup
wait before copying until the non-logged operation are complete.
b) make backup always wait for the non-logged operations to complete and
retake the backup of those files that got affected by the non-logged
operation, if they were already backed up.
c) Some how trigger logging for all the operations after the checkpoint for
the backup until the backup is complete. This one is easy to implement
for non-logged operation that are stated after the backup, but the
tricky case is to trigger logging for those non-logging operation that
started before the backup but are committed during the backup.
3) drop of a table when the file on the disk is being backed up. drop of
a table will result in deletion of the file on the disk, but deletion will get errors
if it is opened for backup.
Some form of synchronization required to make sure that users do not see
weird errors in this case.
4) creating a table/index after the data files are backed up. Basically
recovery system expects that file on the disk exists before the log records
that refer to it are written to the transaction log.
I think roll-forward recovery already handles this case , but should be
tested.
5) data file growth because of inserts when the file(table/index) is being backed up.
Recovery system expects that a page is allocated on the disk
before log records are written to the transaction log about a page to
avoid recovery errors because of space issues except incase of roll-forward
recovery.
I think roll-forward recovery handles this case already; but have to make
sure it will work in this case also. Test cases should be added.
Some form of synchronization is required, to make a stable table snap shot of the
file , if the file is growing when the backup is in progress.
6) checkpoints when the backup is in progress.
I think it not necessary to allow checkpoints when the backup is in
progress. But if some one thinks otherwise , following should
be addressed:
1) make copy of the log control file for the backup before copying any
2) If there are any operations that rely on checkpoint to make the
operation consistent should not be allowed because backup might have
already copied some files when checkpoint happens.
Any comments/suggestions will be appreciated.
Thanks
-suresh
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ØG(" == Øystein Grøvlen (JIRA) <de...@db.apache.org> writes:
ØG(> BaseDataFileFactory.java:
ØG(> * I do think basing which files to back up on the contents of the
ØG(> seg0 directory is very robust.
Typo. Should be 'I do NOT think basing ...'
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> Øystein Grøvlen wrote:
>> Suresh,
>> Just a reminder of some things you promised to fix in the previous
>> review round.
>>
ST> Thanks for reminding. It is on my list. Just thought I can do the
ST> cleanup after I fix the major issues to make online backup work
ST> correctly.
Make sense. Fine.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen wrote:
> Suresh,
>
> Just a reminder of some things you promised to fix in the previous
> review round.
>
Thanks for reminding. It is on my list. Just thought I can do the
cleanup after I fix the major issues to make online backup work
correctly.
Thanks
-suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Suresh,
Just a reminder of some things you promised to fix in the previous
review round.
--
Øystein
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
...
>> RawStore.java:
>> * The BACKUP_FILTER now contains so much, that it would be useful
>> to have a comment that says what is really left to copy.
ST> I agree, comments will be helpful here. I am not sure BACKUP_FILTER
ST> will be be needed any more. Currently it just does not filter JAR dir,
ST> I plan to make JARS also as separate copy after the data segment as
ST> discussed in my previous e-mail.
...
>> * I think it would be helpful with a comment that explained why
>> disableLogArchiveMode() was made synchronized.
ST> Yes, I definitely need to add comments about this synchronization.
ST> This was part of my attempt to prevent parallel backup actions, which
ST> I did not complete as part of this patch.
...
>> RAFContainer.java:
...
>> * privBackupContainer(): - Setting 'done=true' at the start of
>> the method is a bit
>> confusing. I would think another name for this variable
>> would be better.
ST> I will change it to 'backupComplete = false' and reverse the exit
ST> logic, hopefully that will make it more readable.
>> - If an exception is thrown while holding a latch, do one
>> not need to relase the latch?
ST> Yes. Latch has to be released. I will make sure no latches are held
ST> when an error occurs
>> * writeToBackup(): - copies a lot of code from writePage(). One
>> should
>> consider factoring out common code.
ST> I will move the common code into a separate routine.
...
>> LogToFile.java:
...
>> * The javadoc for startLogBackup() says that 'log files are copied
>> after all the data files are backed up, but at the end of the
>> method log files are copied.
>> ReadOnly.java/InputStreamContainer.java: * I do not think javadoc
>> should just be a copy of the
>> interface/superclass, but say something about what is particular
>> to this implementation. In this case, the Javadoc should say
>> that nothing is done.
>> typos:
>> * several occurences of 'backedup'
>> * LogToFile.java: 'eventhough'
>> * RAFContainer.java: 'pahe cache', 'conatiner'
>> * 'Standard Cloudscape error policy'. Should this be changed to
>> Derby?
ST> I will correct the comments.
>> Pet peeves:
>> * Should have Javadoc on all methods, parameters and return
>> values.
>> * Lines longer than 80 chars.
>>
ST> Sound like a good ones to follow. I will add the java doc and fix any
ST> long lines.
ST> Thanks
ST> -suresh
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> I believe Derby gives the OS plenty of opportunity to do parallel I/O
MM> if there are multiple users in the database. Every thread can possibly
MM> be doing I/O at a single time. There may be room to improve in the case
MM> of checkpoint, but not sure how important that is.
See my recent email on Derby I/O issues. Only one thread may
read/write a page from/to a file (table) at a time since readPage and
writePage synchronize their accesses on the RAFContainer object.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Bryan Pendleton <bp...@amberpoint.com>.
> windows allows one to partition in software, I think included in the
> base OS. Can someone say if linux does or not (or at least a particular
> version of linux).
Well, Linux has an extremely powerful component called the Logical
Volume Manager, which sounds like what you mean:
http://www.tldp.org/HOWTO/LVM-HOWTO/index.html
http://www.tldp.org/HOWTO/LVM-HOWTO/whatisvolman.html
This has been widely available in Linux distributions for at
least 5 years or so.
bryan
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
windows allows one to partition in software, I think included in the
base OS. Can someone say if linux does or not (or at least a particular
version of linux). I do agree that hardware can do a more efficient
job than software.
I am not sure how expensive a RAID controller is to anyone who actually
needs multiple disks for performance of their app, I know my < $500
personal home machine came with it.
I believe Derby gives the OS plenty of opportunity to do parallel I/O
if there are multiple users in the database. Every thread can possibly
be doing I/O at a single time. There may be room to improve in the case
of checkpoint, but not sure how important that is.
I agree that single user Derby does not do parallel.
Øystein Grøvlen wrote:
>>>>>>"MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
>
> MM> Yes this is probably incorrect in a number of places. There currently
> MM> is only one segment. The segment number was added in the beginning so
> MM> that if we ever had to do data partitioning ourselves we had a chance
> MM> to
>
> MM> add it without an ugly upgrade. But as you are seeing over time some
> MM> hard coding has happened and if we ever really want another segment it
> MM> will have to be cleaned up.
>
> MM> My opinion is that the OS can do better data partitioning then we can so
> MM> I don't think it is very useful to add that feature.
>
> I agree that the OS often can do a better job with data partitioning
> but that often requires expensive storage systems (e.g., RAID).
>
> I think that many applications would perform better if it was possible
> to put tables and indexes on different disks.
>
> Also, Derby does not give the OS much opportunity to optimize IO as
> long as it does not allow parallel disk accesses to a table.
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "MM" == Mike Matrigali <mi...@sbcglobal.net> writes:
MM> Yes this is probably incorrect in a number of places. There currently
MM> is only one segment. The segment number was added in the beginning so
MM> that if we ever had to do data partitioning ourselves we had a chance
MM> to
MM> add it without an ugly upgrade. But as you are seeing over time some
MM> hard coding has happened and if we ever really want another segment it
MM> will have to be cleaned up.
MM> My opinion is that the OS can do better data partitioning then we can so
MM> I don't think it is very useful to add that feature.
I agree that the OS often can do a better job with data partitioning
but that often requires expensive storage systems (e.g., RAID).
I think that many applications would perform better if it was possible
to put tables and indexes on different disks.
Also, Derby does not give the OS much opportunity to optimize IO as
long as it does not allow parallel disk accesses to a table.
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
comments in line
Suresh Thalamati wrote:
> Hi Øystein,
>
> Thanks for reviewing the patch. My answers are in-line...
>
> Øystein Grøvlen (JIRA) wrote:
>
>> [
>> http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12355620
>> ]
>> Øystein Grøvlen commented on DERBY-239:
>> ---------------------------------------
>>
>> I have looked at the onlinebackup_1.diff patch, and overall the patch
>> looks good. However, I think there is one severe bug in
>> RAFContainer.java. I guess due the cut and paste from writePage(),
>> you are seeking in another file than the one you are writing
>> to (fileData.seek(), but backupRaf.write()).
>>
>
> filedata.seek() in the backup routine is wrong, I will fix it. Good Catch.
>
>> I have a few other comments and questions:
>>
>> General:
>> * If I run backup with this patch, it seems like I will run the
>> new code. Does one not need to change the restore code to be
>> able to restore restore such a backup, or does that the
>> ordinary recovery handle that?
>>
>
> Backup structure is not changed, so current file restore code should
> work ok. Some of the new issues related to online backup are log replay
> issues like, redoing CREATE CONTAINER, new page creation ..etc. I think
> most of these issues are addressed as part of the rollforward recovery.
> I plan to rescan the restore code and do some more testing to find any
> cases that are not handled by the current code.
>
>> RawStore.java:
>> * The BACKUP_FILTER now contains so much, that it would be useful
>> to have a comment that says what is really left to copy.
>
>
> I agree, comments will be helpful here. I am not sure BACKUP_FILTER will
> be be needed any more. Currently it just does not filter JAR dir, I plan
> to make JARS also as separate copy after the data segment as discussed
> in my previous e-mail.
>
>
>> * Intuitively, it seems wrong to hard-code "seg0", but I see that
>> this is done all over the code.
>
>
> Could you please file a JIRA entry for this one. I noticed it too, hard
> coding is definitely not the correct way here. One of us can clean up this.
Yes this is probably incorrect in a number of places. There currently
is only one segment. The segment number was added in the beginning so
that if we ever had to do data partitioning ourselves we had a chance to
add it without an ugly upgrade. But as you are seeing over time some
hard coding has happened and if we ever really want another segment it
will have to be cleaned up.
My opinion is that the OS can do better data partitioning then we can so
I don't think it is very useful to add that feature.
>
>
>> Will there always be only one segment?
>
>
> Yes, at least for now. It is hard wired some where in the access layer I
> think.
>
>
>> What is then the purpose of the segment concept?
>
>
> I think initial design plan was to support some kind of data
> partitioning by storing fragments of tables/indexes on different
> segments or distribute tables into different ssegments. Each segment
> then can be placed on on different disks for performance reasons or to
> avoid derby database size being limited to a single disk/volume.
>
>
>> * backup(File ...) seems like it would create an endless recursion
>> if called. Fortunately, it seems like it never will be
>> called. Why do we need the methods with a File parameter instead
>> of a String. The system procedures uses the String variant.
>
> > Maybe we could just remove the File variant?
>
> I think they are left overs from the old cloudscape versions, File
> variant is not supported any more , these can be cleaned up. There
> should be some File variant calls in access layer also, Could you please
> file Jira for this cleanup, so that these won't forgotten again.
>
>
>> * I think it would be helpful with a comment that explained why
>> disableLogArchiveMode() was made synchronized.
>
>
> Yes, I definitely need to add comments about this synchronization.
> This was part of my attempt to prevent parallel backup actions, which I
> did not complete as part of this patch.
>
>>
>> BaseDataFileFactory.java:
>> * I do think basing which files to back up on the contents of the
>> seg0 directory is very robust. What if someone by accident has
>> written a file with a name that matches the pattern you are
>> looking for. Then I would think you may get a very strange
>> error message that may not be easy to resolve. Could not this
>> be based on some system catalog?
>
>
> I think users should not be creating/deleting any files in SEG0 to start
> with. Pattern matching is just a simple precaution. I thought of
> scanning the catalogs to find the container to backup, but could not
> convince my self that it is required at least for now.
>
> If you think scanning the catalogs is more robust, this enhancement can
> be done at later point of time easily.
Be careful if you go down this route to make sure you pick up the files
that aren't in the catalogs. I know the property conglomerate is not,
there may be others. I think scanning is fine for initial
implementation of online backup.
>
>
>> Another scenario is if someone
>> by accident deletes a file for a table that is not accessed very
>> often. A later backup will then not detect that this file is
>> missing. Since the backup is believed to be succesful, the
>> latest backup of this file may be deleted.
>>
>
> Backup is as good as the database in this case :-) If users access the
> deleted file table from the backup DB , it will fail with NO container
> error, as it will on the main database.
>
> By using the catalog approach also backup can only throw a warning
> about the deleted container. Once the user deletes a container file ,
> there is no way out, users can not even drop that table. I don't think
> making backup fail forever when this scenario occurs will be acceptable
> to the users.
>
>
>> FileContainer.java:
>> * I cannot find any backup-specific about getPageForBackup() so I
>> think a more general name would be better (e.g, getLatchedPage).
>>
>
> Yes, currently this routine does not have any backup specific stuff in
> this routine. I would like to get the page from the cache with specific
> weight for the backup page in future, when the weight based cache
> mechanism is implemented.
>
>
>> RAFContainer.java:
>> * The changes to this file seems not to be quite in line with some
>> of the original design philosophies. I am not sure that is
>> necessarily bad, but it would be nice to here the arguments for
>> doing it this way. More specifically:
>> - While RAFContainer so far has used the
>> StorageRandomAccessFile/StorageFile abstractions, backup
>> use RandomAccessFile/File directly. Is there a particular
>> reason for that?
>
>
> Yes. Currently there is No API available for the users to specify the
> type of Storage Factory to use for the backup. It will be good
> enhancement to add later.
>
> Online backup implementation is doing what the current backup does;
> i.e use the disk IO calls directly , instead of the database Storage
> Factory. I think one reason to do this way is if a database is using
> in-memory Storage Factory, users can backup the database on to a disk.
> If Storage Factory used by the Database is java.io.* , then backup is
> doing the same without going through the Storage Factory interface.
>
>
>> - In order to be able to backup a page, new methods have
>> been added to FileContainer and BasePage/CachedPage, and
>> the RAFContainer is doing latching of pages. This
>> increases coupling to other modules, have alternative
>> designs been considered?
>
>
>
> I was considering doing following minor changes here:
>
> 1) Instead of getting pageData byte array from the Page Object, make
> a request to the page to write it self to backup(like page.backup(....).
>
> 2) Move the backing up and unlatching of the page code to the
> FileContainer.java.
>
> I have not thought of any other better ways to do design this, any ideas
> will be helpful.
>
>
>> * privBackupContainer(): - Setting 'done=true' at the
>> start of the method is a bit
>> confusing. I would think another name for this variable
>> would be better.
>
>
> I will change it to 'backupComplete = false' and reverse the exit logic,
> hopefully that will make it more readable.
>
>> - If an exception is thrown while holding a latch, do one
>> not need to relase the latch?
>
>
> Yes. Latch has to be released. I will make sure no latches are held when
> an error occurs
>
>> * writeToBackup(): - copies a lot of code from
>> writePage(). One should
>> consider factoring out common code.
>
>
> I will move the common code into a separate routine.
>
>> - Due the cut and paste, you are seeking in another file
>> than the one you are writing to. (fileData.seek(), but
>> backupRaf.write())
>>
>
> you are right, I will fix it.
>
>> LogToFile.java:
>> * In getFirstLogNeeded why did you need to change
>> getFirstLogNeeded() to handle a null checkpoint? Is it in case
>> you do backup before any checkpoint has been performed?
>
>
> No. There will be a checkpoint before the backup. I added NULL check
> for testing drop table stub cases without the checkpoint and I also
> noticed a NULL pointer error when shutting down on corrupt cases if a
> checkpoint not occurred by that time. so thought there is no harm in
> adding a NULL check there.
>
>
>> * The javadoc for startLogBackup() says that 'log files are copied
>> after all the data files are backed up, but at the end of the
>> method log files are copied.
>>
>> ReadOnly.java/InputStreamContainer.java: * I do not think javadoc
>> should just be a copy of the
>> interface/superclass, but say something about what is particular
>> to this implementation. In this case, the Javadoc should say
>> that nothing is done.
>>
>> typos:
>> * several occurences of 'backedup'
>> * LogToFile.java: 'eventhough'
>> * RAFContainer.java: 'pahe cache', 'conatiner'
>> * 'Standard Cloudscape error policy'. Should this be changed to
>> Derby?
>
>
>
> I will correct the comments.
>
>
>>
>> Pet peeves:
>> * Should have Javadoc on all methods, parameters and return
>> values.
>> * Lines longer than 80 chars.
>>
>
> Sound like a good ones to follow. I will add the java doc and fix any
> long lines.
>
> Thanks
> -suresh
>
>
>
>
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Øystein Grøvlen wrote:
>>>>>>"ST" == Suresh Thalamati <su...@gmail.com> writes:
>
>
> ST> Hi Øystein,
> ST> Thanks for reviewing the patch. My answers are in-line...
>
> ST> Øystein Grøvlen (JIRA) wrote:
<snip ...>
>
>
> ST> By using the catalog approach also backup can only throw a warning
> ST> about the deleted container. Once the user deletes a container file ,
> ST> there is no way out, users can not even drop that table. I don't think
> ST> making backup fail forever when this scenario occurs will be
> ST> acceptable to the users.
>
> I agree, but detecting in on backup, gives the user the option of
> recovering the table by restoring the previous bakcup.
>
I agree with you, notifying the user as early as possible about the
missing files is a vey good idea. If there is no previous backup then
detecting and stopping the backup is not a good idea either.
I belive this is not common case scenario, this improvements can
be added later, once I get the online backup/restore to work.
<snip ..>
> ST> 2) Move the backing up and unlatching of the page code to the
> ST> FileContainer.java.
>
>
> ST> I have not thought of any other better ways to do design this, any
> ST> ideas will be helpful.
>
> Could the backup code be in a separate class instead of part of
> RAFContainer? It does not seem to use much of RAFContainer's internal
> state.
>
I think backup code in Rafcontainer.java is ok. Purpose of the
backup code there is to backup the container. Backup needs to
synchronize with the container file deletions and truncation. And also
backup code is just couple of methods in this file, so it may not be
worth creating yet another Class file.
Thanks
-suresht
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "ST" == Suresh Thalamati <su...@gmail.com> writes:
ST> Hi Ãystein,
ST> Thanks for reviewing the patch. My answers are in-line...
ST> Ãystein Grøvlen (JIRA) wrote:
...
>> * Intuitively, it seems wrong to hard-code "seg0", but I see that
>> this is done all over the code.
ST> Could you please file a JIRA entry for this one. I noticed it too,
ST> hard coding is definitely not the correct way here. One of us can
ST> clean up this.
Done. Derby-664.
...
>> * backup(File ...) seems like it would create an endless recursion
>> if called. Fortunately, it seems like it never will be
>> called. Why do we need the methods with a File parameter instead
>> of a String. The system procedures uses the String variant.
>> Maybe we could just remove the File variant?
ST> I think they are left overs from the old cloudscape versions, File
ST> variant is not supported any more , these can be cleaned up. There
ST> should be some File variant calls in access layer also, Could you
ST> please file Jira for this cleanup, so that these won't forgotten
ST> again.
Done. Derby-665.
...
>> BaseDataFileFactory.java:
>> * I do think basing which files to back up on the contents of the
>> seg0 directory is very robust. What if someone by accident has
>> written a file with a name that matches the pattern you are
>> looking for. Then I would think you may get a very strange
>> error message that may not be easy to resolve. Could not this
>> be based on some system catalog?
ST> I think users should not be creating/deleting any files in SEG0 to
ST> start with. Pattern matching is just a simple precaution. I thought of
ST> scanning the catalogs to find the container to backup, but could not
ST> convince my self that it is required at least for now.
I agree that users should not mess with seg0. However, I think part
of being an easy-to-use database, is to be able to give meaningful
error messages when users mess things up.
ST> If you think scanning the catalogs is more robust, this enhancement
ST> can be done at later point of time easily.
Ok. I will see if I feel the itch. 8-)
>> Another scenario is if someone
>> by accident deletes a file for a table that is not accessed very
>> often. A later backup will then not detect that this file is
>> missing. Since the backup is believed to be succesful, the
>> latest backup of this file may be deleted.
>>
ST> Backup is as good as the database in this case :-) If users access the
ST> deleted file table from the backup DB , it will fail with NO container
ST> error, as it will on the main database.
ST> By using the catalog approach also backup can only throw a warning
ST> about the deleted container. Once the user deletes a container file ,
ST> there is no way out, users can not even drop that table. I don't think
ST> making backup fail forever when this scenario occurs will be
ST> acceptable to the users.
I agree, but detecting in on backup, gives the user the option of
recovering the table by restoring the previous bakcup.
>> FileContainer.java:
>> * I cannot find any backup-specific about getPageForBackup() so I
>> think a more general name would be better (e.g, getLatchedPage).
>>
ST> Yes, currently this routine does not have any backup specific stuff in
ST> this routine. I would like to get the page from the cache with
ST> specific weight for the backup page in future, when the weight based
ST> cache mechanism is implemented.
In that case, I would suggest that the weight be a parameter to the
method, not part of the name.
>> RAFContainer.java:
>> * The changes to this file seems not to be quite in line with some
>> of the original design philosophies. I am not sure that is
>> necessarily bad, but it would be nice to here the arguments for
>> doing it this way. More specifically:
>> - While RAFContainer so far has used the
>> StorageRandomAccessFile/StorageFile abstractions, backup
>> use RandomAccessFile/File directly. Is there a particular
>> reason for that?
ST> Yes. Currently there is No API available for the users to specify the
ST> type of Storage Factory to use for the backup. It will be good
ST> enhancement to add later.
ST> Online backup implementation is doing what the current backup does;
ST> i.e use the disk IO calls directly , instead of the database Storage
ST> Factory. I think one reason to do this way is if a database is using
ST> in-memory Storage Factory, users can backup the database on to a
ST> disk. If Storage Factory used by the Database is java.io.* , then
ST> backup is doing the same without going through the Storage Factory
ST> interface.
Makes sense. If one want a StorageFactory for backup, one can add that
later.
>> - In order to be able to backup a page, new methods have
>> been added to FileContainer and BasePage/CachedPage, and
>> the RAFContainer is doing latching of pages. This
>> increases coupling to other modules, have alternative
>> designs been considered?
ST> I was considering doing following minor changes here:
ST> 1) Instead of getting pageData byte array from the Page Object, make
ST> a request to the page to write it self to backup(like page.backup(....).
Sounds good to me. Then much the same structure is used for backup as
for clean.
ST> 2) Move the backing up and unlatching of the page code to the
ST> FileContainer.java.
ST> I have not thought of any other better ways to do design this, any
ST> ideas will be helpful.
Could the backup code be in a separate class instead of part of
RAFContainer? It does not seem to use much of RAFContainer's internal
state.
...
--
Øystein
Re: [jira] Commented: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Suresh Thalamati <su...@gmail.com>.
Hi Øystein,
Thanks for reviewing the patch. My answers are in-line...
Øystein Grøvlen (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12355620 ]
>
> Øystein Grøvlen commented on DERBY-239:
> ---------------------------------------
>
> I have looked at the onlinebackup_1.diff patch, and overall the patch
> looks good. However, I think there is one severe bug in
> RAFContainer.java. I guess due the cut and paste from writePage(),
> you are seeking in another file than the one you are writing
> to (fileData.seek(), but backupRaf.write()).
>
filedata.seek() in the backup routine is wrong, I will fix it. Good Catch.
> I have a few other comments and questions:
>
> General:
> * If I run backup with this patch, it seems like I will run the
> new code. Does one not need to change the restore code to be
> able to restore restore such a backup, or does that the
> ordinary recovery handle that?
>
Backup structure is not changed, so current file restore code should
work ok. Some of the new issues related to online backup are log
replay issues like, redoing CREATE CONTAINER, new page creation ..etc.
I think most of these issues are addressed as part of the rollforward
recovery. I plan to rescan the restore code and do some more testing
to find any cases that are not handled by the current code.
> RawStore.java:
> * The BACKUP_FILTER now contains so much, that it would be useful
> to have a comment that says what is really left to copy.
I agree, comments will be helpful here. I am not sure BACKUP_FILTER
will be be needed any more. Currently it just does not filter JAR dir,
I plan to make JARS also as separate copy after the data segment as
discussed in my previous e-mail.
> * Intuitively, it seems wrong to hard-code "seg0", but I see that
> this is done all over the code.
Could you please file a JIRA entry for this one. I noticed it too,
hard coding is definitely not the correct way here. One of us can
clean up this.
> Will there always be only one segment?
Yes, at least for now. It is hard wired some where in the access layer
I think.
> What is then the purpose of the segment concept?
I think initial design plan was to support some kind of data
partitioning by storing fragments of tables/indexes on different
segments or distribute tables into different ssegments. Each segment
then can be placed on on different disks for performance reasons or to
avoid derby database size being limited to a single disk/volume.
> * backup(File ...) seems like it would create an endless recursion
> if called. Fortunately, it seems like it never will be
> called. Why do we need the methods with a File parameter instead
> of a String. The system procedures uses the String variant.
> Maybe we could just remove the File variant?
I think they are left overs from the old cloudscape versions, File
variant is not supported any more , these can be cleaned up. There
should be some File variant calls in access layer also, Could you
please file Jira for this cleanup, so that these won't forgotten again.
> * I think it would be helpful with a comment that explained why
> disableLogArchiveMode() was made synchronized.
Yes, I definitely need to add comments about this synchronization.
This was part of my attempt to prevent parallel backup actions, which
I did not complete as part of this patch.
>
> BaseDataFileFactory.java:
> * I do think basing which files to back up on the contents of the
> seg0 directory is very robust. What if someone by accident has
> written a file with a name that matches the pattern you are
> looking for. Then I would think you may get a very strange
> error message that may not be easy to resolve. Could not this
> be based on some system catalog?
I think users should not be creating/deleting any files in SEG0 to
start with. Pattern matching is just a simple precaution. I thought of
scanning the catalogs to find the container to backup, but could not
convince my self that it is required at least for now.
If you think scanning the catalogs is more robust, this enhancement
can be done at later point of time easily.
> Another scenario is if someone
> by accident deletes a file for a table that is not accessed very
> often. A later backup will then not detect that this file is
> missing. Since the backup is believed to be succesful, the
> latest backup of this file may be deleted.
>
Backup is as good as the database in this case :-) If users access
the deleted file table from the backup DB , it will fail with NO
container error, as it will on the main database.
By using the catalog approach also backup can only throw a warning
about the deleted container. Once the user deletes a container file ,
there is no way out, users can not even drop that table. I don't think
making backup fail forever when this scenario occurs will be
acceptable to the users.
> FileContainer.java:
> * I cannot find any backup-specific about getPageForBackup() so I
> think a more general name would be better (e.g, getLatchedPage).
>
Yes, currently this routine does not have any backup specific stuff
in this routine. I would like to get the page from the cache with
specific weight for the backup page in future, when the weight based
cache mechanism is implemented.
> RAFContainer.java:
> * The changes to this file seems not to be quite in line with some
> of the original design philosophies. I am not sure that is
> necessarily bad, but it would be nice to here the arguments for
> doing it this way. More specifically:
> - While RAFContainer so far has used the
> StorageRandomAccessFile/StorageFile abstractions, backup
> use RandomAccessFile/File directly. Is there a particular
> reason for that?
Yes. Currently there is No API available for the users to specify the
type of Storage Factory to use for the backup. It will be good
enhancement to add later.
Online backup implementation is doing what the current backup does;
i.e use the disk IO calls directly , instead of the database Storage
Factory. I think one reason to do this way is if a database is using
in-memory Storage Factory, users can backup the database on to a disk.
If Storage Factory used by the Database is java.io.* , then backup is
doing the same without going through the Storage Factory interface.
> - In order to be able to backup a page, new methods have
> been added to FileContainer and BasePage/CachedPage, and
> the RAFContainer is doing latching of pages. This
> increases coupling to other modules, have alternative
> designs been considered?
I was considering doing following minor changes here:
1) Instead of getting pageData byte array from the Page Object, make
a request to the page to write it self to backup(like page.backup(....).
2) Move the backing up and unlatching of the page code to the
FileContainer.java.
I have not thought of any other better ways to do design this, any
ideas will be helpful.
> * privBackupContainer():
> - Setting 'done=true' at the start of the method is a bit
> confusing. I would think another name for this variable
> would be better.
I will change it to 'backupComplete = false' and reverse the exit
logic, hopefully that will make it more readable.
> - If an exception is thrown while holding a latch, do one
> not need to relase the latch?
Yes. Latch has to be released. I will make sure no latches are held
when an error occurs
> * writeToBackup():
> - copies a lot of code from writePage(). One should
> consider factoring out common code.
I will move the common code into a separate routine.
> - Due the cut and paste, you are seeking in another file
> than the one you are writing to. (fileData.seek(), but
> backupRaf.write())
>
you are right, I will fix it.
> LogToFile.java:
> * In getFirstLogNeeded why did you need to change
> getFirstLogNeeded() to handle a null checkpoint? Is it in case
> you do backup before any checkpoint has been performed?
No. There will be a checkpoint before the backup. I added NULL
check for testing drop table stub cases without the checkpoint and I
also noticed a NULL pointer error when shutting down on corrupt cases
if a checkpoint not occurred by that time. so thought there is no harm
in adding a NULL check there.
> * The javadoc for startLogBackup() says that 'log files are copied
> after all the data files are backed up, but at the end of the
> method log files are copied.
>
> ReadOnly.java/InputStreamContainer.java:
> * I do not think javadoc should just be a copy of the
> interface/superclass, but say something about what is particular
> to this implementation. In this case, the Javadoc should say
> that nothing is done.
>
> typos:
> * several occurences of 'backedup'
> * LogToFile.java: 'eventhough'
> * RAFContainer.java: 'pahe cache', 'conatiner'
> * 'Standard Cloudscape error policy'. Should this be changed to
> Derby?
I will correct the comments.
>
> Pet peeves:
> * Should have Javadoc on all methods, parameters and return
> values.
> * Lines longer than 80 chars.
>
Sound like a good ones to follow. I will add the java doc and fix any
long lines.
Thanks
-suresh
[jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Øystein Grøvlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=comments#action_12355620 ]
Øystein Grøvlen commented on DERBY-239:
---------------------------------------
I have looked at the onlinebackup_1.diff patch, and overall the patch
looks good. However, I think there is one severe bug in
RAFContainer.java. I guess due the cut and paste from writePage(),
you are seeking in another file than the one you are writing
to (fileData.seek(), but backupRaf.write()).
I have a few other comments and questions:
General:
* If I run backup with this patch, it seems like I will run the
new code. Does one not need to change the restore code to be
able to restore restore such a backup, or does that the
ordinary recovery handle that?
RawStore.java:
* The BACKUP_FILTER now contains so much, that it would be useful
to have a comment that says what is really left to copy.
* Intuitively, it seems wrong to hard-code "seg0", but I see that
this is done all over the code. Will there always be only one
segment? What is then the purpose of the segment concept?
* backup(File ...) seems like it would create an endless recursion
if called. Fortunately, it seems like it never will be
called. Why do we need the methods with a File parameter instead
of a String. The system procedures uses the String variant.
Maybe we could just remove the File variant?
* I think it would be helpful with a comment that explained why
disableLogArchiveMode() was made synchronized.
BaseDataFileFactory.java:
* I do think basing which files to back up on the contents of the
seg0 directory is very robust. What if someone by accident has
written a file with a name that matches the pattern you are
looking for. Then I would think you may get a very strange
error message that may not be easy to resolve. Could not this
be based on some system catalog? Another scenario is if someone
by accident deletes a file for a table that is not accessed very
often. A later backup will then not detect that this file is
missing. Since the backup is believed to be succesful, the
latest backup of this file may be deleted.
FileContainer.java:
* I cannot find any backup-specific about getPageForBackup() so I
think a more general name would be better (e.g, getLatchedPage).
RAFContainer.java:
* The changes to this file seems not to be quite in line with some
of the original design philosophies. I am not sure that is
necessarily bad, but it would be nice to here the arguments for
doing it this way. More specifically:
- While RAFContainer so far has used the
StorageRandomAccessFile/StorageFile abstractions, backup
use RandomAccessFile/File directly. Is there a particular
reason for that?
- In order to be able to backup a page, new methods have
been added to FileContainer and BasePage/CachedPage, and
the RAFContainer is doing latching of pages. This
increases coupling to other modules, have alternative
designs been considered?
* privBackupContainer():
- Setting 'done=true' at the start of the method is a bit
confusing. I would think another name for this variable
would be better.
- If an exception is thrown while holding a latch, do one
not need to relase the latch?
* writeToBackup():
- copies a lot of code from writePage(). One should
consider factoring out common code.
- Due the cut and paste, you are seeking in another file
than the one you are writing to. (fileData.seek(), but
backupRaf.write())
LogToFile.java:
* In getFirstLogNeeded why did you need to change
getFirstLogNeeded() to handle a null checkpoint? Is it in case
you do backup before any checkpoint has been performed?
* The javadoc for startLogBackup() says that 'log files are copied
after all the data files are backed up, but at the end of the
method log files are copied.
ReadOnly.java/InputStreamContainer.java:
* I do not think javadoc should just be a copy of the
interface/superclass, but say something about what is particular
to this implementation. In this case, the Javadoc should say
that nothing is done.
typos:
* several occurences of 'backedup'
* LogToFile.java: 'eventhough'
* RAFContainer.java: 'pahe cache', 'conatiner'
* 'Standard Cloudscape error policy'. Should this be changed to
Derby?
Pet peeves:
* Should have Javadoc on all methods, parameters and return
values.
* Lines longer than 80 chars.
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup_1.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (DERBY-239) Need a online backup feature that
does not block update operations when online backup is in progress.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
committed this fix as svn 329934.
Suresh Thalamati (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
>
> Suresh Thalamati updated DERBY-239:
> -----------------------------------
>
> Attachment: onlinebackup_2.diff
>
> Fix to the problem found by Øystein while reviewing the previous online backup patch(online_backup1.diff).
> Backup of a container code was doing a seek incorrectly on the file container instead of the backup file.
>
> Tests: All tests passed on jdk142 Windows XP.
>
> It would be great if some one can commit this patch.
>
>
> Thanks
> -suresht
>
>
>
>>Need a online backup feature that does not block update operations when online backup is in progress.
>>--------------------------------------------------------------------------------------------------------
>>
>> Key: DERBY-239
>> URL: http://issues.apache.org/jira/browse/DERBY-239
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Versions: 10.1.1.0
>> Reporter: Suresh Thalamati
>> Assignee: Suresh Thalamati
>> Attachments: onlinebackup.html, onlinebackup_1.diff, onlinebackup_2.diff
>>
>>Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
>>Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
>>backup is in the progress on the server.
>
>
[jira] Updated: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Posted by "Suresh Thalamati (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-239?page=all ]
Suresh Thalamati updated DERBY-239:
-----------------------------------
Attachment: onlinebackup_2.diff
Fix to the problem found by Øystein while reviewing the previous online backup patch(online_backup1.diff).
Backup of a container code was doing a seek incorrectly on the file container instead of the backup file.
Tests: All tests passed on jdk142 Windows XP.
It would be great if some one can commit this patch.
Thanks
-suresht
> Need a online backup feature that does not block update operations when online backup is in progress.
> --------------------------------------------------------------------------------------------------------
>
> Key: DERBY-239
> URL: http://issues.apache.org/jira/browse/DERBY-239
> Project: Derby
> Type: New Feature
> Components: Store
> Versions: 10.1.1.0
> Reporter: Suresh Thalamati
> Assignee: Suresh Thalamati
> Attachments: onlinebackup.html, onlinebackup_1.diff, onlinebackup_2.diff
>
> Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
> Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
> backup is in the progress on the server.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira