You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Ole Solberg (JIRA)" <ji...@apache.org> on 2008/03/20 20:41:25 UTC

[jira] Created: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Number of log files (and log dir size) on the slave increases continuously
--------------------------------------------------------------------------

                 Key: DERBY-3562
                 URL: https://issues.apache.org/jira/browse/DERBY-3562
             Project: Derby
          Issue Type: Bug
          Components: Replication
    Affects Versions: 10.4.0.0, 10.5.0.0
         Environment: -
            Reporter: Ole Solberg
         Attachments: master_slave-db_size-6.jpg

I did a simple test inserting tuples in a table during replication:

The attached file 'master_slave-db_size-6.jpg' shows that 
the size of the log directory (and number of files in the log directory)
increases continuously during replication, while on master the size 
(and number of files) never exceeds ~12Mb (12 files?) in this scenario.

The seg0 directory on the slave stays at the same size as the master 
seg0 directory.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Ole Solberg (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ole Solberg updated DERBY-3562:
-------------------------------

    Attachment: master_slave-db_size-6.jpg

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>         Attachments: master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "V.Narayanan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581024#action_12581024 ] 

V.Narayanan commented on DERBY-3562:
------------------------------------

Not sure if the analysis here points at the rate of increase in the slave or the increase
in the slave relative to the master. 

But isn't an increase in the log size of a slave expected, because the Slave receives log 
records from the master and use LogToFile#appendLogRecord to write these to disk. 
So basically the slave uses the logs received from the master to recover and move to a 
transaction consistent state.

Incase you have not seen the following points from Derby-3071 (Modify logging subsystem for slave replication mode)

* SlaveFactory (DERBY-3021) will receive log records from the master and use LogToFile#appendLogRecord to write these to disk. While in slave mode, only SlaveFactory will be allowed to append log records.
* The thread running LogToFile#recover will recover (redo) one log file at a time (like now), but will not be allowed to open a log file X until that file is no longer being written to. Thus, while appenLogFile writes to logX.dat, recover will be allowed to read all log files up to and including logX-1.dat but will then have to wait until appendLogRecord starts writing to logX+1.dat.

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>         Attachments: master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jørgen Løland updated DERBY-3562:
---------------------------------

    Attachment: derby-3562-1a.stat
                derby-3562-1a.diff

Patch 1a deletes unnecessary, old log files on the slave when a checkpoint is encountered in the log received from the master. All tests (except those failing tinderbox) passed, including the replication test suite.

Requesting review.

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>            Assignee: Jørgen Løland
>         Attachments: derby-3562-1a.diff, derby-3562-1a.stat, master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581046#action_12581046 ] 

Øystein Grøvlen commented on DERBY-3562:
----------------------------------------

I think the issue may be that checkpointing is only happening at the master.  Hence, log files are garbage-collected on the master, but not on the slave.  If that is the case, one risk that with a long-running replication, the disk on the slave side will fill up.  I do not think this is acceptable, and we need to look into how the recovery thread can be modified to garbage collect log files when checkpoint log records are encountered.

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>         Attachments: master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581816#action_12581816 ] 

Jørgen Løland commented on DERBY-3562:
--------------------------------------

> I do not think this is acceptable, and we need to look into how the recovery thread can be modified to garbage collect log files when checkpoint log records are encountered.

I agree. This has to be fixed. 

I think it will be possible to use the checkpoints received from the master to discard old log files on the slave. The checkpoint contains a low water mark pointing to the oldest log record that needs to be kept. Log older than the LWM should be discardable on the slave as well. Need some more investigation, though...

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>         Attachments: master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582949#action_12582949 ] 

Jørgen Løland commented on DERBY-3562:
--------------------------------------

Mike Matrigali writes:
> I didn't quite follow all of this, and admit i am not up on replication.
> It would be nice if this process was the exact code as the normal
> checkpoint processing.  So a checkpoint would be triggered and then
> after it had done it's work it would do the appropriate cleanup.  If you
> do the cleanup too soon then redo recovery of the slave won't work - is
> that expected to work or at that point to you just restart from scratch
> from master.

> The existing code that replay's multiple checkpoints may be wierd as it
> may assume that this is recovery of a backed up database that is meant
> to keep all of it's log files.  Make sure to not break that.

> Is there a concept of a "fully" recoverable slave, ie. one that is
> supposed to keep all of it's log files so that it is recoverable in
> case of a data crash.  As I said may not be necessary as there is
> always the master.  Just good to know what is expected.
Mike,

Thank you for expressing your concerns. I'll do my best to explain why I think the proposed solution will work.

The patch adds functionality to the checkpoint processing used during recovery (LogToFile#checkpointInRFR). During recovery, the dirty data pages are flushed to disk, and the log.ctrl file is updated to point to the new checkpoint currently being processed.

With the patch [1], the log files that are older than the currently processed checkpoint's Undo Low Water Mark (undo LWM) are then deleted. The undo LWM points to the earliest log record that may be required to do recovery [2]. Since the log files are processed sequentially and the data pages have been flushed, the undo LWM in the checkpoint is equally valid during recovery (aka slave replication mode) as during normal transaction processing.

Once replication has successfully started, the slave database will always be recoverable [3], but not in case of corrupted data blocks [4]. You may at any time crash the Derby serving the slave database and then reboot it. The used-to-be-slave database will then recover to a transaction consistent state including the modifications from all transactions whose commit log record was written to disk on the slave before the crash.

Please follow up if you think I may have misunderstood anything or did not answer your questions good enough.

[1] The patch only applies to slave replication mode. Backup is not affected as to not break the "fully" recoverability feature for backups.
[2] The first log record of the oldest transaction in the checkpoint's transaction table.
[3] If "fully" recoverable means recovering in presence of corrupted data blocks, this is currently not supported for replication.
[4] Not including jar files, as explained in DERBY-3552.

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>            Assignee: Jørgen Løland
>         Attachments: derby-3562-1a.diff, derby-3562-1a.stat, master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Re: [jira] Assigned: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by Mike Matrigali <mi...@sbcglobal.net>.
I didn't quite follow all of this, and admit i am not up on replication.
It would be nice if this process was the exact code as the normal
checkpoint processing.  So a checkpoint would be triggered and then
after it had done it's work it would do the appropriate cleanup.  If you
do the cleanup too soon then redo recovery of the slave won't work - is
that expected to work or at that point to you just restart from scratch
from master.

The existing code that replay's multiple checkpoints may be wierd as it
may assume that this is recovery of a backed up database that is meant
to keep all of it's log files.  Make sure to not break that.

Is there a concept of a "fully" recoverable slave, ie. one that is
supposed to keep all of it's log files so that it is recoverable in
case of a data crash.  As I said may not be necessary as there is
always the master.  Just good to know what is expected.


Jørgen Løland (JIRA) wrote:
>      [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> 
> Jørgen Løland reassigned DERBY-3562:
> ------------------------------------
> 
>     Assignee: Jørgen Løland
> 
>> Number of log files (and log dir size) on the slave increases continuously
>> --------------------------------------------------------------------------
>>
>>                 Key: DERBY-3562
>>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>>             Project: Derby
>>          Issue Type: Bug
>>          Components: Replication
>>    Affects Versions: 10.4.0.0, 10.5.0.0
>>         Environment: -
>>            Reporter: Ole Solberg
>>            Assignee: Jørgen Løland
>>         Attachments: master_slave-db_size-6.jpg
>>
>>
>> I did a simple test inserting tuples in a table during replication:
>> The attached file 'master_slave-db_size-6.jpg' shows that 
>> the size of the log directory (and number of files in the log directory)
>> increases continuously during replication, while on master the size 
>> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
>> The seg0 directory on the slave stays at the same size as the master 
>> seg0 directory.
> 



[jira] Assigned: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jørgen Løland reassigned DERBY-3562:
------------------------------------

    Assignee: Jørgen Løland

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>            Assignee: Jørgen Løland
>         Attachments: master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Øystein Grøvlen resolved DERBY-3562.
------------------------------------

       Resolution: Fixed
    Fix Version/s: 10.5.0.0
                   10.4.1.0
       Derby Info:   (was: [Patch Available])

Thanks for the patch Jørgen.  I have verified that log files now get garbage collected at slave.
Merged to trunk at revision 643336.
Merged to 10.4 branch at revision 643892.

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>            Assignee: Jørgen Løland
>             Fix For: 10.4.1.0, 10.5.0.0
>
>         Attachments: derby-3562-1a.diff, derby-3562-1a.stat, master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jørgen Løland updated DERBY-3562:
---------------------------------

    Derby Info: [Patch Available]

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>            Assignee: Jørgen Løland
>         Attachments: derby-3562-1a.diff, derby-3562-1a.stat, master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (DERBY-3562) Number of log files (and log dir size) on the slave increases continuously

Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/DERBY-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581946#action_12581946 ] 

Jørgen Løland commented on DERBY-3562:
--------------------------------------

When a checkpoint operation is encountered on the slave, cached data blocks are synced (flushed to disk). The log.ctrl file is then updated to point to this cp. It appears that the only thing needed is to call truncateLog with the low water mark of the cp. Manual tests confirm this (failover succeeds on the slave and old log files are removed continuously) , but I haven't run all the tests yet. 

> Number of log files (and log dir size) on the slave increases continuously
> --------------------------------------------------------------------------
>
>                 Key: DERBY-3562
>                 URL: https://issues.apache.org/jira/browse/DERBY-3562
>             Project: Derby
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 10.4.0.0, 10.5.0.0
>         Environment: -
>            Reporter: Ole Solberg
>            Assignee: Jørgen Løland
>         Attachments: master_slave-db_size-6.jpg
>
>
> I did a simple test inserting tuples in a table during replication:
> The attached file 'master_slave-db_size-6.jpg' shows that 
> the size of the log directory (and number of files in the log directory)
> increases continuously during replication, while on master the size 
> (and number of files) never exceeds ~12Mb (12 files?) in this scenario.
> The seg0 directory on the slave stays at the same size as the master 
> seg0 directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.