[RDF Patch] Looking at Talis Changesets and other proposals (v2)

[Andy Seaborne <an...@apache.org>]

Revised notes on Talis Changesets:


==== RDF Patch compared to Talis Changesets.

Talis Changesets (TCS) are defined by:

http://docs.api.talis.com/getting-started/changesets
http://docs.api.talis.com/getting-started/changeset-protocol
http://vocab.org/changeset/schema.html

== Brief Description

A Changeset is a set of triples to remove and a set of triples to add, 
recorded as a single RDF graph.  There is a fixed "subject of change" - 
a changeset is a change to a single resource.  The triples of the change 
must all have the same subject and this must be the subject of change.

The triples of the change are recorded as reified statements.  This is 
necessary so that triples can be grouped into removal and addition sets. 
The change set can have descriptive information about the change. 
Because the changset is an RDF graph, the graph can say who was the 
creator, record the reason for the change, and the date the modification 
was created (not executed).  This also requires that the change triples 
are reified.

ChangeSet can be linked together to produce a sequence of changes.  This 
is how to get changes to several resources - a list of changesets.

== Pros and Cons

This approach a some advantages and some disadvantages:
(some of these can be overcome by fairly obvious changes to the 
definitions).

1/ Changes relate only to one resource.

You can't make a coordinated set of changes, such as adding a batch of 
several new resources in a single HTTP request.

2/ Blank nodes can't be handled.

There is no way give the subject of change if it is a blank node nor to 
link to existing blank nodes in the data. (The Talis platform didn't 
support blank nodes.)

3/ Scale issues : streaming and latency.

Streaming is valuable at a scale because changes can start to be applied 
as the data arrive, rather than buffering the changes until the end of 
change is seen, then applying the changes.

For large changes, this also impacts latency. Doing some or all of the 
changes as data arrives gets an overlap in processing between sender and 
receiver.

However, the use of a RDF graph as a changeset blocks streaming.

It needs the whole changeset graph to be available before any changes 
are made.  The whole graph is needed to validate the changeset (e.g. all 
reified triples have the same subject), and order of triples in a 
serialization of a graph is arbitrary (esp. if produced  by a generic 
RDF serializer) so, for example, the "subject of change" triple could be 
last, of the additions and removals can be mixed in any order.  To get 
stable changes, it is necessary to have a rule like all removals done 
before the additions are done.

This is a limitation at scale. In practice, a changeset must be parsed 
into memory (standard parser), validated (changeset specific code) and 
applied (changeset specific code).  The design can't support streaming 
nor changes which may be larger than available RAM (e.g. millions of 
triples).

It does mean that a standard RDF tool kit can be used to produce the 
change set (with suitable application code to build the graph structure) 
and to parse it at the receiver, together with some application code for 
producing, validating and executing a changeset.

4/ The feature of metadata per change is a useful feature.

5/ Change sets only work for a change to a resource in a single graph.

== Other

Graph literals:

Some other proposals have been made (like Delta, or variants based on 
TriG) where named graphs are used instead of reified triples.   The 
scaling issue remains - processing can't start until the whole change 
has been seen.

Delta:

{ ?x bank:accountNo "1234578"}
   diff:deletion { ?x bank:balance 4000};
   diff:insertion { ?x bank:balance 3575}
}

Restricted SPARQL Update:

That leaves a restricted SPARQL update: e.g. DELETE DATA, INSERT DATA, 
and maybe DELETE WHERE.

As soon as additional restrictions apply, then, to validate, you need a 
special parser so the primary advantage of reusing of existing tools) is 
only partial.