You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Rob Vesse <rv...@dotnetrdf.org> on 2015/11/12 01:42:12 UTC

TDB/Lizard/TDB 2 Replication

Andy

I was talking to some folks at a major bank the other day about TDB and got
an interesting question that I didn't have an answer to.  Essentially they
were interested in learning how you would provide replication and hot-standy
with TDB

For current generation TDB I told them that people typically place a Load
Balancer in front on multiple TDB instances for read-centric workloads and
manually handle replicating updates to all systems (usually be temporarily
disabling the services, applying the updates and restarting)

However for Lizard and TDB 2 I did not really know what the answer would be?
Lizard is obviously designed as a fault-tolerant system but I don't know if
the same can be said for TDB 2 (which I understand more to be about changing
the on-disk data structures)

For bonus points they were also interested in whether Lizard/TDB 2 would
offer customisable replication policies?  For example they would like to be
able to configure some replicas to be read-only while having others be
read/write?

Rob



Re: TDB/Lizard/TDB 2 Replication

Posted by Andy Seaborne <an...@apache.org>.
On 12/11/15 17:55, Rob Vesse wrote:
> I think they expected all the replicas to stay roughly consistent but
> they'd like to have a set up where some users are directed to specific
> replicas (they were thinking of geographically distributed replicas) and
> that users in certain geographic reasons would only be able to read the
> data and not to write it.  So the system as a whole would be read/write
> but individual replicas might be exposed as read-only services.
>
> Is that something that is doable?

Very interesting use case.  I'd approach it as change propagation 
problem of a federation of systems, rather than a single system of 
replicated machines. Geographically distributed + single system is hard 
work.

One way is to capture transaction logs and propagate the logs - they 
need to be applied in order.  RDF patch, enhanced with begin/commit 
boundaries would work really well for this - maybe in a binary version.

The systems could even keep their blank nodes in step.  It even works 
for API made changes.

In fact, this is something TDB2 needs.  There, while compressing the 
database, it needs to capture any transaction changes to replay them.

A related approach is to propagate SPARQL updates and play them (again, 
in order) at each node. This isn't quite so general. The blank nodes are 
different blank nodes, and it does not cover the graph store protocol 
(though obviously that could be added easily enough), nor API (can't see 
how to do blank nodes).

I can also imagine a third setup by using named graphs for data 
management. If they are small enough, propagate graphs with the graph 
store protocol.

I like the first because it establishes a building block for many things.

Having a dataset that recorded all its changes would be good for rolling 
back in time.  Or sign commits with who made them.

	Andy







Re: TDB/Lizard/TDB 2 Replication

Posted by Stephen Allen <sa...@apache.org>.
On Fri, Nov 13, 2015 at 7:21 AM, Andy Seaborne <an...@apache.org> wrote:

> On 13/11/15 07:28, Stephen Allen wrote:
> ...
>
>
> This doesn't really solve the replica/hot standby issues you're
>> discussing.  But if you are interested in high availability and failover
>> for a single TDB instance, you might take a look at DRBD [1][2].  Although
>> I've mentioned it before, I still haven't had a chance to use it.  It does
>> seem to offer a neat solution for a "warm" standby with automatic failover
>> (using Heartbeat).
>>
>> -Stephen
>>
>> [1] http://drbd.linbit.com/
>> [2] https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device
>>
>> DRBD could work for a (very) warm standby.  If the transaction log can be
> sync'ed across the machines, then it's zero loss and looks quite promising.
>
> For live systems, do you know how it interacts with memory mapped files
> and also whether the apps can be notified of changes for managing their
> in-process caches.
>
>         Andy
>
>
I believe it acts just like a regular block device, and doesn't know
anything about memory.  So it'll replicate disk writes, and appears like it
can be run in both synchronous and asynchronous modes (but for a database
presumably you'd probably pay the performance penalty for synchronous
writes to ensure durability guarantees).

I was assuming that the replica machine would not start up its TDB process
until it was notified of failover, otherwise its internal state
(memory-mapped files and caches) would be out of sync.

Having a true hot standby with read capabilities does not appear to be
possible with DRBD.  Even if it were possible to update memory-mapped files
on the replica and notify the app of changes, it seems to me that you would
need more than that.  Either application support for distributed
transactions (complex) or copy-on-write with no garbage collection
(inefficient).  Otherwise the master wouldn't know when readers
transactions on the slave are finished.

-Stephen

Re: TDB/Lizard/TDB 2 Replication

Posted by Andy Seaborne <an...@apache.org>.
On 13/11/15 07:28, Stephen Allen wrote:
...


> This doesn't really solve the replica/hot standby issues you're
> discussing.  But if you are interested in high availability and failover
> for a single TDB instance, you might take a look at DRBD [1][2].  Although
> I've mentioned it before, I still haven't had a chance to use it.  It does
> seem to offer a neat solution for a "warm" standby with automatic failover
> (using Heartbeat).
>
> -Stephen
>
> [1] http://drbd.linbit.com/
> [2] https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device
>
DRBD could work for a (very) warm standby.  If the transaction log can 
be sync'ed across the machines, then it's zero loss and looks quite 
promising.

For live systems, do you know how it interacts with memory mapped files 
and also whether the apps can be notified of changes for managing their 
in-process caches.

	Andy


Re: TDB/Lizard/TDB 2 Replication

Posted by Stephen Allen <sa...@apache.org>.
On Thu, Nov 12, 2015 at 12:55 PM, Rob Vesse <rv...@dotnetrdf.org> wrote:

> Comments inline:
>
> On 12/11/2015 06:31, "Andy Seaborne" <an...@apache.org> wrote:
>
> >On 12/11/15 00:42, Rob Vesse wrote:
> >> Andy
> >>
> >> I was talking to some folks at a major bank the other day about TDB and
> >>got
> >> an interesting question that I didn't have an answer to.  Essentially
> >>they
> >> were interested in learning how you would provide replication and
> >>hot-standy
> >> with TDB
> >>
> >> For current generation TDB I told them that people typically place a
> >>Load
> >> Balancer in front on multiple TDB instances for read-centric workloads
> >>and
> >> manually handle replicating updates to all systems (usually be
> >>temporarily
> >> disabling the services, applying the updates and restarting)
> >
> >There is a tradeoff - another way, avoiding loss of service during
> >updates is keep the service running and accept a window of inconsistency
> >and update each server live (in parallel or in sequence for slight
> >different effects).  Load balancers are (usually) sticky so the nature
> >of the inconsistency is reduced for each client. Depending on the use
> >case where the read-workload and publisher workload are separate,
> >perfect consistency may not matter.
> >
> >> However for Lizard and TDB 2 I did not really know what the answer
> >>would be?
> >> Lizard is obviously designed as a fault-tolerant system but I don't
> >>know if
> >> the same can be said for TDB 2 (which I understand more to be about
> >>changing
> >> the on-disk data structures)
> >
> >TDB2 is TDB built and avoids the limitations on transaction size.  TDB2
> >itself is a single machine system like TDB.
> >
> >If you have any feedback to express that more clearly than
> >
> >http://mail-archives.apache.org/mod_mbox/jena-dev/201506.mbox/%3C5575B7B3
> .
> >8020101%40apache.org%3E
> >
> >please let me know.   I'm too close to the tech!
>
> No that makes perfect sense
>
> My understanding was that TDB primarily solved the transaction and disk
> usage growth issues
>
> >
> >
> >I have been thinking that packaging Lizard as a two server system ("two"
> >=> small number, each a complete DB replica so able to operate
> >independently).  This is distributed transactions (multi-master, with
> >the one true active transaction restriction which is really the same as
> >master-slave with a master election for each transaction) across
> >multiple TDB2 instances.
> >
> >> For bonus points they were also interested in whether Lizard/TDB 2 would
> >> offer customisable replication policies?  For example they would like
> >>to be
> >> able to configure some replicas to be read-only while having others be
> >> read/write?
> >
> >Yes, architecturally. No, in implementation.
> >
> >And the middle ground of update some relicas synchronously and some
> >async after the transaction has formally committed.  (i.e. 5 replicas,
> >update 3 now and 2 "later").
> >
> >The current implementation is "update all replicas".  It's missing the
> >async catch-up aspect and what do do with crashes during catch-up.
> >
> >Is that what was meant by "some read-only"?
>
> No I don't think so
>
> >
> >Or was it a partition of the dataset with some parts read-only and on
> >servers that reflect that and some parts read-write?
>
> I think they expected all the replicas to stay roughly consistent but
> they'd like to have a set up where some users are directed to specific
> replicas (they were thinking of geographically distributed replicas) and
> that users in certain geographic reasons would only be able to read the
> data and not to write it.  So the system as a whole would be read/write
> but individual replicas might be exposed as read-only services.
>
> Is that something that is doable?
>
> Rob
>
> >
> >I guess it comes down to what happens if an update happens - do
> >read-only systems eventually catch up? Are they unaffected because
> >updates didn't apply to their part of the graphs in the datasets? Other
> >meaning?
> >
>

This doesn't really solve the replica/hot standby issues you're
discussing.  But if you are interested in high availability and failover
for a single TDB instance, you might take a look at DRBD [1][2].  Although
I've mentioned it before, I still haven't had a chance to use it.  It does
seem to offer a neat solution for a "warm" standby with automatic failover
(using Heartbeat).

-Stephen

[1] http://drbd.linbit.com/
[2] https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device

Re: TDB/Lizard/TDB 2 Replication

Posted by Rob Vesse <rv...@dotnetrdf.org>.
Comments inline:

On 12/11/2015 06:31, "Andy Seaborne" <an...@apache.org> wrote:

>On 12/11/15 00:42, Rob Vesse wrote:
>> Andy
>>
>> I was talking to some folks at a major bank the other day about TDB and
>>got
>> an interesting question that I didn't have an answer to.  Essentially
>>they
>> were interested in learning how you would provide replication and
>>hot-standy
>> with TDB
>>
>> For current generation TDB I told them that people typically place a
>>Load
>> Balancer in front on multiple TDB instances for read-centric workloads
>>and
>> manually handle replicating updates to all systems (usually be
>>temporarily
>> disabling the services, applying the updates and restarting)
>
>There is a tradeoff - another way, avoiding loss of service during
>updates is keep the service running and accept a window of inconsistency
>and update each server live (in parallel or in sequence for slight
>different effects).  Load balancers are (usually) sticky so the nature
>of the inconsistency is reduced for each client. Depending on the use
>case where the read-workload and publisher workload are separate,
>perfect consistency may not matter.
>
>> However for Lizard and TDB 2 I did not really know what the answer
>>would be?
>> Lizard is obviously designed as a fault-tolerant system but I don't
>>know if
>> the same can be said for TDB 2 (which I understand more to be about
>>changing
>> the on-disk data structures)
>
>TDB2 is TDB built and avoids the limitations on transaction size.  TDB2
>itself is a single machine system like TDB.
>
>If you have any feedback to express that more clearly than
>
>http://mail-archives.apache.org/mod_mbox/jena-dev/201506.mbox/%3C5575B7B3.
>8020101%40apache.org%3E
>
>please let me know.   I'm too close to the tech!

No that makes perfect sense

My understanding was that TDB primarily solved the transaction and disk
usage growth issues

>
>
>I have been thinking that packaging Lizard as a two server system ("two"
>=> small number, each a complete DB replica so able to operate
>independently).  This is distributed transactions (multi-master, with
>the one true active transaction restriction which is really the same as
>master-slave with a master election for each transaction) across
>multiple TDB2 instances.
>
>> For bonus points they were also interested in whether Lizard/TDB 2 would
>> offer customisable replication policies?  For example they would like
>>to be
>> able to configure some replicas to be read-only while having others be
>> read/write?
>
>Yes, architecturally. No, in implementation.
>
>And the middle ground of update some relicas synchronously and some
>async after the transaction has formally committed.  (i.e. 5 replicas,
>update 3 now and 2 "later").
>
>The current implementation is "update all replicas".  It's missing the
>async catch-up aspect and what do do with crashes during catch-up.
>
>Is that what was meant by "some read-only"?

No I don't think so

>
>Or was it a partition of the dataset with some parts read-only and on
>servers that reflect that and some parts read-write?

I think they expected all the replicas to stay roughly consistent but
they'd like to have a set up where some users are directed to specific
replicas (they were thinking of geographically distributed replicas) and
that users in certain geographic reasons would only be able to read the
data and not to write it.  So the system as a whole would be read/write
but individual replicas might be exposed as read-only services.

Is that something that is doable?

Rob

>
>I guess it comes down to what happens if an update happens - do
>read-only systems eventually catch up? Are they unaffected because
>updates didn't apply to their part of the graphs in the datasets? Other
>meaning?
>
>	Andy
>
>>
>> Rob
>>
>





Re: TDB/Lizard/TDB 2 Replication

Posted by Andy Seaborne <an...@apache.org>.
On 12/11/15 00:42, Rob Vesse wrote:
> Andy
>
> I was talking to some folks at a major bank the other day about TDB and got
> an interesting question that I didn't have an answer to.  Essentially they
> were interested in learning how you would provide replication and hot-standy
> with TDB
>
> For current generation TDB I told them that people typically place a Load
> Balancer in front on multiple TDB instances for read-centric workloads and
> manually handle replicating updates to all systems (usually be temporarily
> disabling the services, applying the updates and restarting)

There is a tradeoff - another way, avoiding loss of service during 
updates is keep the service running and accept a window of inconsistency 
and update each server live (in parallel or in sequence for slight 
different effects).  Load balancers are (usually) sticky so the nature 
of the inconsistency is reduced for each client. Depending on the use 
case where the read-workload and publisher workload are separate, 
perfect consistency may not matter.

> However for Lizard and TDB 2 I did not really know what the answer would be?
> Lizard is obviously designed as a fault-tolerant system but I don't know if
> the same can be said for TDB 2 (which I understand more to be about changing
> the on-disk data structures)

TDB2 is TDB built and avoids the limitations on transaction size.  TDB2 
itself is a single machine system like TDB.

If you have any feedback to express that more clearly than

http://mail-archives.apache.org/mod_mbox/jena-dev/201506.mbox/%3C5575B7B3.8020101%40apache.org%3E

please let me know.   I'm too close to the tech!


I have been thinking that packaging Lizard as a two server system ("two" 
=> small number, each a complete DB replica so able to operate 
independently).  This is distributed transactions (multi-master, with 
the one true active transaction restriction which is really the same as 
master-slave with a master election for each transaction) across 
multiple TDB2 instances.

> For bonus points they were also interested in whether Lizard/TDB 2 would
> offer customisable replication policies?  For example they would like to be
> able to configure some replicas to be read-only while having others be
> read/write?

Yes, architecturally. No, in implementation.

And the middle ground of update some relicas synchronously and some 
async after the transaction has formally committed.  (i.e. 5 replicas, 
update 3 now and 2 "later").

The current implementation is "update all replicas".  It's missing the 
async catch-up aspect and what do do with crashes during catch-up.

Is that what was meant by "some read-only"?

Or was it a partition of the dataset with some parts read-only and on 
servers that reflect that and some parts read-write?

I guess it comes down to what happens if an update happens - do 
read-only systems eventually catch up? Are they unaffected because 
updates didn't apply to their part of the graphs in the datasets? Other 
meaning?

	Andy

>
> Rob
>