You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Mike Boom <mb...@verimatrix.com> on 2008/07/17 22:35:19 UTC
Replication
I hope this is an appropriate place to ask this question. Is there a
plan in the future for replication to provide readable replicas, or even
better multiple masters? If so what kind of time line are we looking at
for completion? Thanks.
Re: Replication
Posted by Rick Hillegas <Ri...@Sun.COM>.
Hi Mike,
Narayanan has already sent a pointer to the main replication JIRA:
http://issues.apache.org/jira/browse/Derby-2872 Besides Narayanan, the
other major contributors to that effort were Jørgen Løland and Øystein
Grøvlen.
Regards,
-Rick
Mike Boom wrote:
> I may want to add master/master replication or at least master/readonly
> replication. How would I find out who the original contributors would
> be?
>
> -----Original Message-----
> From: Richard.Hillegas@Sun.COM [mailto:Richard.Hillegas@Sun.COM]
> Sent: Monday, July 21, 2008 12:42 PM
> To: derby-dev@db.apache.org
> Subject: Re: Replication
>
> Mike Boom wrote:
>
>> I hope this is an appropriate place to ask this question. Is there a
>> plan in the future for replication to provide readable replicas, or
>> even better multiple masters? If so what kind of time line are we
>> looking at for completion? Thanks.
>>
>>
> Hi Mike,
>
> I have not seen a lot of work on replication recently, other than
> stabilization of the existing functionality and tests. Probably, the
> original contributors would be happy to coach someone who wanted to
> extend Derby replication.
>
> Regards,
> -Rick
>
>
Re: Replication
Posted by Mike Matrigali <mi...@sbcglobal.net>.
V Narayanan wrote:
> Hi Mike,
>
> Currently the replication functionality is not a pluggable module but is
> rather strongly linked with the store
> layer. For example LogToFile has been actually modified to catch the log
> records and store them in the log
> buffer in addition to storing them on the master. If you are thinking
> about implementing a synchronous or
> a synchronous (but read-only masters, I think in this case also you
> would be constrained to commit a transaction
> only after all the masters sync), you would do great to think of how
> this can be implemented as a separate module
> rather than coupling it tightly with the store as is happening now.
>
> You could also think about convincing the community about the advantages
> a synchronous replication scheme might
> bring to Derby over the existing asynchronous scheme and probably think
> about modifying existing implementation
> to work in a sychronous way, or atleast think about how the existing
> functionality can be bundled as a separate unit.
I would also suggest looking at adding a new replication module rather
than change the existing code. The system can be set up such that the
new replication module can extend the existing functionality to achieve
the additional work necessary. It was how I had expected a hot standby
to be implemented.
I think a read only replicant could be implemented using a lot of
existing functionality. With
changes that did:
1) add replication module(s)
2) module to extend log buffer writes in some way such that log records
are sent to read only replicant some how. How and how often depend on
how current you need the read only replicant. You could probaby even
do this with no in line changes to log writing and just read the log
files from disk along with assuring the db is in backup mode which
always keeps all log files. This mode would be interesting as it would
inpact the master performance as little as possible (except for possible
I/O throughput to log device).
3) add new module that applies log records and thus changes to the db,
but does not allow any external changes. To support read committed or
higher than some extra work may need to be done in this area to get
expected locks while applying log records. Currently the code that does
this assumes it is boot time and need not get locks. Basically this
code wants to be a copy of the code that is done during boot recovery
but handle applying the log records "online" vs. "offline". This is
probably the most challenging bit.
>
> Some good places to start would be
>
> http://issues.apache.org/jira/browse/Derby-2872
>
> http://wiki.apache.org/db-derby/ReplicationWriteup
>
> You could also look at some synchronous replication products listed here
>
> http://wiki.apache.org/db-derby/UsesOfDerby
>
> looking at them would probably help you what type of an end product you
> would expect.
>
> There was an effort from a student to provide hot standby functionality
> to derby, you can find the paper and related work here
>
> https://issues.apache.org/jira/browse/DERBY-2852
>
> I will help you in whatever way I can in this effort, but I may not
> actually be operation in synchronous mode ;) , I promise
> asynchronous read-only, commit-maybe, support to your effort :) .
>
> But your proposal looks interesting, please do remember to keep the
> community informed about your ideas, efforts and problems
> constantly. I have noticed that the community appreciates continuous
> feedback on issues.
>
> Narayanan
> Mike Boom wrote:
>> I may want to add master/master replication or at least master/readonly
>> replication. How would I find out who the original contributors would
>> be?
>>
>> -----Original Message-----
>> From: Richard.Hillegas@Sun.COM [mailto:Richard.Hillegas@Sun.COM]
>> Sent: Monday, July 21, 2008 12:42 PM
>> To: derby-dev@db.apache.org
>> Subject: Re: Replication
>>
>> Mike Boom wrote:
>>
>>> I hope this is an appropriate place to ask this question. Is there a
>>> plan in the future for replication to provide readable replicas, or
>>> even better multiple masters? If so what kind of time line are we
>>> looking at for completion? Thanks.
>>>
>>>
>> Hi Mike,
>>
>> I have not seen a lot of work on replication recently, other than
>> stabilization of the existing functionality and tests. Probably, the
>> original contributors would be happy to coach someone who wanted to
>> extend Derby replication.
>>
>> Regards,
>> -Rick
>>
>>
>
Re: Replication
Posted by V Narayanan <V....@Sun.COM>.
Hi Mike,
Currently the replication functionality is not a pluggable module but is
rather strongly linked with the store
layer. For example LogToFile has been actually modified to catch the log
records and store them in the log
buffer in addition to storing them on the master. If you are thinking
about implementing a synchronous or
a synchronous (but read-only masters, I think in this case also you
would be constrained to commit a transaction
only after all the masters sync), you would do great to think of how
this can be implemented as a separate module
rather than coupling it tightly with the store as is happening now.
You could also think about convincing the community about the advantages
a synchronous replication scheme might
bring to Derby over the existing asynchronous scheme and probably think
about modifying existing implementation
to work in a sychronous way, or atleast think about how the existing
functionality can be bundled as a separate unit.
Some good places to start would be
http://issues.apache.org/jira/browse/Derby-2872
http://wiki.apache.org/db-derby/ReplicationWriteup
You could also look at some synchronous replication products listed here
http://wiki.apache.org/db-derby/UsesOfDerby
looking at them would probably help you what type of an end product you
would expect.
There was an effort from a student to provide hot standby functionality
to derby, you can find the paper and related work here
https://issues.apache.org/jira/browse/DERBY-2852
I will help you in whatever way I can in this effort, but I may not
actually be operation in synchronous mode ;) , I promise
asynchronous read-only, commit-maybe, support to your effort :) .
But your proposal looks interesting, please do remember to keep the
community informed about your ideas, efforts and problems
constantly. I have noticed that the community appreciates continuous
feedback on issues.
Narayanan
Mike Boom wrote:
> I may want to add master/master replication or at least master/readonly
> replication. How would I find out who the original contributors would
> be?
>
> -----Original Message-----
> From: Richard.Hillegas@Sun.COM [mailto:Richard.Hillegas@Sun.COM]
> Sent: Monday, July 21, 2008 12:42 PM
> To: derby-dev@db.apache.org
> Subject: Re: Replication
>
> Mike Boom wrote:
>
>> I hope this is an appropriate place to ask this question. Is there a
>> plan in the future for replication to provide readable replicas, or
>> even better multiple masters? If so what kind of time line are we
>> looking at for completion? Thanks.
>>
>>
> Hi Mike,
>
> I have not seen a lot of work on replication recently, other than
> stabilization of the existing functionality and tests. Probably, the
> original contributors would be happy to coach someone who wanted to
> extend Derby replication.
>
> Regards,
> -Rick
>
>
RE: Replication
Posted by Mike Boom <mb...@verimatrix.com>.
I may want to add master/master replication or at least master/readonly
replication. How would I find out who the original contributors would
be?
-----Original Message-----
From: Richard.Hillegas@Sun.COM [mailto:Richard.Hillegas@Sun.COM]
Sent: Monday, July 21, 2008 12:42 PM
To: derby-dev@db.apache.org
Subject: Re: Replication
Mike Boom wrote:
>
> I hope this is an appropriate place to ask this question. Is there a
> plan in the future for replication to provide readable replicas, or
> even better multiple masters? If so what kind of time line are we
> looking at for completion? Thanks.
>
Hi Mike,
I have not seen a lot of work on replication recently, other than
stabilization of the existing functionality and tests. Probably, the
original contributors would be happy to coach someone who wanted to
extend Derby replication.
Regards,
-Rick
Re: Replication
Posted by Rick Hillegas <Ri...@Sun.COM>.
Mike Boom wrote:
>
> I hope this is an appropriate place to ask this question. Is there a
> plan in the future for replication to provide readable replicas, or
> even better multiple masters? If so what kind of time line are we
> looking at for completion? Thanks.
>
Hi Mike,
I have not seen a lot of work on replication recently, other than
stabilization of the existing functionality and tests. Probably, the
original contributors would be happy to coach someone who wanted to
extend Derby replication.
Regards,
-Rick