You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Jørgen Løland (JIRA)" <ji...@apache.org> on 2007/07/25 13:26:31 UTC
[jira] Created: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Replication: Add a ReplicationMaster controller that will manage replication on the master side
-----------------------------------------------------------------------------------------------
Key: DERBY-2977
URL: https://issues.apache.org/jira/browse/DERBY-2977
Project: Derby
Issue Type: Sub-task
Components: Services
Affects Versions: 10.4.0.0
Reporter: Jørgen Løland
The replication master role includes many tasks:
* set up a network connection to the slave
* sending the database to the slave before starting replication
* make sure that log records are appended to the network buffer, and that the log is later sent to the slave
* etc
This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Derby Info: [Patch Available]
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: repli_logbuffer_v2b.diff
repli_logbuffer_v2b.stat
Thanks for the review, Øystein. I have attached a new patch, v2b, addressing all your comments except 2:
1. Changed constructor of LogAccessFile to call LogToFile#initLogAccessFileReplication. I cannot think of a better name than startReplicationMasterRole, but feel free to suggest one...
9. Do you mean LogToFile#start/stop? If so, startReplicationMasterRole uses this.masterFactory because a parameter has the same name, which is not the case for stopReplicationMasterRole.
Not fixed:
2. To be honest, I don't like either of these strategies ;-)
With the current strategy:
next() returns true -> some info is available - could be a log file switch, could be a log record
next() returns false -> done
This allows the loop:
while (next()) {
if (scan.isFileSwitch())
switchLogFile();
else
applyLog(scan.getData());
}
The suggested strategy:
next() returns true -> log available, maybe also a log file switch
next() returns false -> no log available, but maybe a log file switch?
I'm ok with the "true" scenario, but not the "false" scenario. I.e., I don't think it is ok for ReplicationLogScan to have vital information (a log switch *must* be performed at the same place on the slave and master) when it returns false. I did not change this in the attached patch. Ideally, I would like to see a third strategy that was better than both these, but cannot think of
one.
All tests pass cleanly.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12523522 ]
Jørgen Løland commented on DERBY-2977:
--------------------------------------
I plan to implement the ReplicationMaster Controller incrementally in multiple steps:
1) Add the basic code to Derby so that the master controller can be booted as a service when a "startmaster" command is issued to NetworkServerControl
2-n) Incrementally add controller logic as replication patches are added to Derby.
The current plan for step 1 is:
* NetworkServerControl will lookup the correct database, and call a startReplicationMaster method on that database service. This call will propagate through the database and access layers to RawStore.java.
* RawStore#startReplicationMaster will call Monitor#bootServiceModule on the MasterFactory (interface for MasterController)
* When booted, the MasterController service will initialize some basic variables but do nothing fancy.
Comments are appreciated.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Derby Info: [Patch Available]
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530689 ]
Jørgen Løland commented on DERBY-2977:
--------------------------------------
See DERBY-2872 for version 2 of the proof of concept.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515285 ]
Jørgen Løland commented on DERBY-2977:
--------------------------------------
The initial plan for the replication master controller:
The replication master controller is started when the "start replication master" command (see functional specification in
DERBY-2872) is issued for a database. There will be exactly one replication controller instance for every database with a master role.
It seems to be a good idea to boot the replication controller as a service module on a per need basis. This is, e.g., done to boot the TransactionFactory and DataFactory services in engine/org/apache/derby/impl/store/raw/RawStore.java
RawStore.java#boot, line 215-221:
-----8<-----
xactFactory = (TransactionFactory)
Monitor.bootServiceModule(
create, this, getTransactionFactoryModule(), properties);
dataFactory = (DataFactory)
Monitor.bootServiceModule(
create, this, getDataFactoryModule(), properties);
----->8-----
The module will need an API for these commands:
start replication
stop replication
The controller will be managing the replication strategy (asynchronous, 1-safe, 2-safe, very-safe). For now, only
asynchronous is considered. If more strategies are added later, the API should probably be extended with methods to dynamically change the replication strategy.
Development on this issue will likely be incremental - the service must be modified as more functionality is in place.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530954 ]
Øystein Grøvlen commented on DERBY-2977:
----------------------------------------
The patch looks very good. I have a few comments, but most are minor nits:
1. LogToFile:
It seems a bit strange to add checks for replicationMasterMode in
situations where replication should not be going on (e.g., during
recovery). Would it be better to add a method to LogToFile that
the LogAccessFile constructor could use to check whether
replication was going on? Or maybe pass it as a parameter to the
constructor? With the current usage, I would change the name of
LogAccessFile#startReplicationMasterRole since it is not called
only when replication is started.
2. ReplicationLogScan:
I am not quite sure I like the way you handle the mix of log
records and log file switch records. I wonder whether it would be
cleaner if when a log file switch is encountered, one would set the
logFileSwitch flag and continue to read the next log record. Then,
if next() returns true there will always be a valid log record, but
you will have to check for log file switch before you handle it.
3. ReplicationLogScan, javadoc:
Why does the @see list the same class twice?
4. ReplicationLogScan#next, comment in exception handling (not new):
Typo: (currupt)
5. ReplicationLogBuffer, javadoc (Not added by this patch):
I feel "... consists of n LogBufferElement, each of which can store
m log records ..." is a bit confusing since it sounds like the
number of log records in a buffer (m) is a configurable constant.
6. ReplicationLogBuffer#appendBytes:
You do not want javadoc for the params?
7. MasterController:
Import of ReplicationMessage is unused.
8. MasterController/MasterFactory#appendLogRecord, javadoc:
Mismatch between javadoc and parameter name for length.
9. MasterController#startReplicationMasterRole/stopReplicationMasterRole:
Why is one of the member fields prefixed with 'this.', but not the
other?
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: derby_2977_1.diff
derby_2977_1.stat
Attaching a patch, v1, intended to solve step 1 described above.
The patch adds a service (MasterFactory (interface), MasterController) that will be used to manage replication on the master side. Currently, the service does very little work except write out a few "MasterController started" lines. These printouts will be removed when code is added to the service.
The service is started by calling Database#startReplicationMaster on the database that will be replicated. However, there is no code calling this method yet, so the patch is currently not used for anything. The intention is that command line calls to NetworkServerControl will start the service. DERBY-2954 has been added for that task.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524499 ]
Øystein Grøvlen commented on DERBY-2977:
----------------------------------------
I think this patch looks very good. Most of my comments are minor:
1. Why do you need to expose LogToFile#logBufferSize outside the
class? The Logger used by transactions handles the buffer size
dynamically, why cannot this be done in this case, too?
2. MasterFactory#flushedTo: The Javadoc should say something
about why this method is needed and when it will be called.
3. I think it would be good if it is were more clear from the
MasterController code, which methods were implementation of which
interface. E.g., have separate sections of the file for each of
the interfaces with a heading in a comment.
4. MasterController#canSupport: The javadoc for the return value
should say something about what it means, not just its
current implementation.
5. ReadOnly: The new import does not seem necessary.
6. MasterFactory javadoc says "This is an implementation ...", but it
is not. It is the interface. MasterController is the
implementation. I also think the note about being far from
complete belongs in the class and not the interface. (There is
also a missing "to" in the second sentence of the javadoc.)
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Øystein Grøvlen updated DERBY-2977:
-----------------------------------
Derby Info: (was: [Patch Available])
Fix Version/s: 10.4.0.0
Committed patch, repli_logbuffer_v2d.diff, as revision 585900.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Fix For: 10.4.0.0
>
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat, repli_logbuffer_v2d.diff, repli_logbuffer_v2d.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland reassigned DERBY-2977:
------------------------------------
Assignee: Jørgen Løland
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Component/s: (was: Services)
Replication
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Replication
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Fix For: 10.4.0.0
>
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat, repli_logbuffer_v2d.diff, repli_logbuffer_v2d.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Øystein Grøvlen updated DERBY-2977:
-----------------------------------
Derby Info: (was: [Patch Available])
Committed patch derby_2977_1c.stat.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: repli_logbuffer_v2d.diff
repli_logbuffer_v2d.stat
Thanks for reviewing patch v2c Øystein.
In patch v2d, I left the implementation of your 2nd comment as is since you don't seem to feel very strongly about it. The other issues have been fixed. All tests passed.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat, repli_logbuffer_v2d.diff, repli_logbuffer_v2d.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524680 ]
Øystein Grøvlen commented on DERBY-2977:
----------------------------------------
Jørgen Løland (JIRA) wrote:
> Thanks for reviewing the patch, Øystein.
>
> The attached patch, v1b, addresses your comments in the following way:
>
>> 1. Why do you need to expose LogToFile#logBufferSize outside the
>> class? The Logger used by transactions handles the buffer size
>> dynamically, why cannot this be done in this case, too?
>
> Actually, as far as I can tell, the log buffer used by
> transactions (LogAccessFile) uses the same LogToFile#buffersize
> exposed here. Is there another file I should look at? I use this
> buffer size because I cannot think of any other size that would
> be better in the general case. I can change this if you have any
> suggestions for an appropriate size. If so, MasterFactory should
> probably store the default size as a static final variable. Left
> unchanged in v1b since I can't think of a better alternative.
Sorry, I thought there was one buffer per log record, which is similar
to FileLogger which has a dynamic buffer size. Still, I do not think
that what is the optimal buffer size for writing to disk is
necessarily the best buffer size when sending messages over the
network. I would think that should be independent of each other.
The rest of the changes looks very good, but I have one question on
flushedTo: It seems to be the assumption that a log record should not
be sent to the slave before it is written to disk? Is this a
necessary requirement? If yes, maybe that should be documented too.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: repli_logbuffer_v2c.stat
repli_logbuffer_v2c.diff
Patch v2c is a minor modification to v2b (added some javadoc and a simple log instant change for log chunks in LogAccessFile). It replaces v2b. All tests passed
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland closed DERBY-2977.
--------------------------------
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Replication
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Fix For: 10.4.0.0
>
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat, repli_logbuffer_v2d.diff, repli_logbuffer_v2d.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: derby_2977_1b.diff
derby_2977_1b.stat
Thanks for reviewing the patch, Øystein.
The attached patch, v1b, addresses your comments in the following way:
> 1. Why do you need to expose LogToFile#logBufferSize outside the
> class? The Logger used by transactions handles the buffer size
> dynamically, why cannot this be done in this case, too?
Actually, as far as I can tell, the log buffer used by
transactions (LogAccessFile) uses the same LogToFile#buffersize
exposed here. Is there another file I should look at? I use this
buffer size because I cannot think of any other size that would
be better in the general case. I can change this if you have any
suggestions for an appropriate size. If so, MasterFactory should
probably store the default size as a static final variable. Left
unchanged in v1b since I can't think of a better alternative.
2-6: fixed in v1b.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535534 ]
Øystein Grøvlen commented on DERBY-2977:
----------------------------------------
Thanks for addressing my comments in the new patch, Jørgen. I have
just a few minor follow-up comments (same numbering as in my previous
review):
1. Your basically looks good. IMHO, calling the new method
'checkForReplication' or something instead of
'initLogAccessFileReplication' since it will not always init
replication. (Check is also the main verb you use to describe in
the javadoc what it is doing.)
As an alternative to 'startReplicationMasterRole', I suggest
'setReplicationMasterRole', since as I said, it is not always
starting replication.
2. A third alternative implementation that will behave almost the same
as the current strategy as seen from the user, is to remember any
logFileSwitch at the end of a log chunk until the next chunk
arrives, and let switchLogFile() be true for the first log record
of the next chunk. Then, removing 'else' will be the only necessary change
to your example code.
4. As far as I can see, you have not addressed this typo.
9. I think I meant LogAccessFile#stopReplicationMasterRole. Anyway,
it is fixed now.
10. (NEW) To me, adding instant as a parameter to
LogAccessFile#writeToLog seems like a better solution than
maintaining a field 'instantCurrentChunk'.
None of the above issues are very important, and it is your
call whether you would like to fix any of them before I commit.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: derby_2977_1c.diff
derby_2977_1c.stat
Thanks for the review of the patch, Øystein.
Attaching a patch, v1c, addressing the buffer size issue. I agree that the replication buffer size should not depend on the disk buffer size. In lack of a better size, I still use 32k, but now defined in a constant in MasterController. This size will probably be subject to changes once we are able to test replication performance :)
Regarding flushedTo: I do not see any good reason why we should not allow log records to be sent to the slave before these are written to disk. Any operations that are sent prematurely to the slave will be aborted if a commit has not been received on the slave. The disk buffer also works this way - it may write log to disk even when force has not been called.
There is, however, a possibility that the following sequence happens during commit processing on the master:
1. Commit log record is appended to disk buffer
2. Commit log record is appended to replication buffer
3. Replication log shipping thread gets CPU time and ships all buffered log to the slave
4. Master crashes before commit log record has been safely written to disk.
This would lead to a case where the commit log record is not on disk on the master, but is on the slave. However, I do not think this is a problem because:
* Commit has been requested in the first place, so the client should agree that committing was the intention.
* For the operations of that transaction to be seen by anyone connecting to the previous slave, the failover command must be issued on the slave. The functional specification states that when this happens, the database that had the master role is no longer the active database. Thus, the actual disk image of the crashed master database is not interesting.
I have started the tests, but expect no failures since the code in the patch is not in use. I will revert with the results as soon as the tests complete.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland resolved DERBY-2977.
----------------------------------
Resolution: Fixed
Only remaining task is to ship the database from the master to the slave location using the replication network at replication startup. Filed DERBY-3434 for this.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Replication
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Fix For: 10.4.0.0
>
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat, repli_logbuffer_v2b.diff, repli_logbuffer_v2b.stat, repli_logbuffer_v2c.diff, repli_logbuffer_v2c.stat, repli_logbuffer_v2d.diff, repli_logbuffer_v2d.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Øystein Grøvlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524733 ]
Øystein Grøvlen commented on DERBY-2977:
----------------------------------------
Thanks Jørgen. What you say makes sense. I have started the tests, and I will commit the latest patch tomorrow if the tests run well.
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2977) Replication: Add a ReplicationMaster
controller that will manage replication on the master side
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jørgen Løland updated DERBY-2977:
---------------------------------
Attachment: repli_logbuffer_v2.diff
repli_logbuffer_v2.stat
Version 2 of the proof of concept code showed a problem with the way log is appended to the repliation buffer. Currently, LogToFile#appendLogRecord is used to write log to the replication master log buffer. However, the LogAccessFile class that wraps write operations to log file also write checksums. We need these checksums at the slave because without them, the log instants on the slave and master will differ, in turn making recovery impossible (undo operations refer to a specific instant, which will not be correct).
The attached patch, repli_logbuffer_v2.*, changes this erroneous behavior. With this patch, log is appended to the repliation log buffer in LogAccessFile, not LogToFile like now. The patch modifies the following files:
M java/engine/org/apache/derby/impl/services/replication/buffer/LogBufferElement.java
M java/engine/org/apache/derby/impl/services/replication/buffer/ReplicationLogBuffer.java
The buffer now accepts chunks of log records from LogAccessFile#writeToLog (i.e., appended to replication buffer in the same method that writes the log to disk) instead of single log records from LogToFile#appendLogRecord
M java/engine/org/apache/derby/impl/services/replication/slave/ReplicationLogScan.java
Modified the slave-side log parser to read the new chunk of log record format
M java/engine/org/apache/derby/impl/services/replication/master/MasterController.java
M java/engine/org/apache/derby/iapi/services/replication/master/MasterFactory.java
Modified appendLog signature to accept log from LogAccessFile instead of from LogToFile
M java/engine/org/apache/derby/impl/store/raw/log/LogAccessFile.java
Can be set inReplicationMasterMode, in which log is appended to MasterFactory
M java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
Removed the code that appends log to replication log buffer, and tells LogAccessFile to go into replication master mode when needed.
Allsuites and derbyall completed without error
> Replication: Add a ReplicationMaster controller that will manage replication on the master side
> -----------------------------------------------------------------------------------------------
>
> Key: DERBY-2977
> URL: https://issues.apache.org/jira/browse/DERBY-2977
> Project: Derby
> Issue Type: Sub-task
> Components: Services
> Affects Versions: 10.4.0.0
> Reporter: Jørgen Løland
> Assignee: Jørgen Løland
> Attachments: derby_2977_1.diff, derby_2977_1.stat, derby_2977_1b.diff, derby_2977_1b.stat, derby_2977_1c.diff, derby_2977_1c.stat, repli_logbuffer_v2.diff, repli_logbuffer_v2.stat
>
>
> The replication master role includes many tasks:
> * set up a network connection to the slave
> * sending the database to the slave before starting replication
> * make sure that log records are appended to the network buffer, and that the log is later sent to the slave
> * etc
> This issue is for adding a controller that will start/stop/initiate all services needed for the replication master role.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.