Batched update

[Andy Seaborne <an...@apache.org>]

JENA-528

== History

The BulkUpdateHandler interface is supposed to help system optimize 
large(r) scale addition and deletion of triple (it's graph-centric so 
datasets don't feature here; it predates datasets).

The API contract is that a collection of triples can be added or be 
deleted in a single operation.  There is no possibilities of looking at 
the data while a bulk operation is in progress (its a single API call). 
  The default implementation of SimpleBulkUpdateHandler turned these 
into a loop.

This contract has problems: how big is a batch? does it have to reside 
in memory? (the nearest to avoiding that is a pull-style passing in an 
Iterator but that is hard to use).

ARP (the RDF/XML parser) did use batching.  It builds an array of 1000 
triples and sends them into a model in such batches.  Other parsers did 
not; they do can signal start/end of parse run though via a different 
mechanism.

= RDB

The original work was triggered by RDB.

The gain was because a single DB transaction could be used for multiple 
additions.  Without any other information, the current state of the JDBC 
connection would be used, and that could be autocommit which has severe 
limitations.

= SDB

SDB does have a bulk loader - it manipulates temporary tables in the 
database to accumulate and manipulate the data to insert (e.g. 
de-duplication).  It also gets away from the JDBC/autocommit issue.

The temporary tables are not always transaction safe (depends on the
DB).  SDB even copes with bulk deletion (but I'm unclear about what 
happens if a a triple is inserted and deleted in the same batch as 
inserts and deletes are treated as two separate groups so interleaving 
is lost).

A batch is ideally in the 10k-20K range.  These batches accumulate - not 
a single API operation.

Batching is done internally (even if the client app did).  It is 
triggered by explicit SDB-specific calls or graph events 
GraphEvents.startRead and GraphEvents.finishRead.  These can be called 
manually by the application; they cause the appropriate GraphSDB 
operations to be called and these can be called explicitly as well.

Batched additions can't be seen by Graph.find during the batch update.

http://jena.apache.org/documentation/sdb/loading_data.html

The bulk triple API operations were overlaid on this mechanism.

= TDB

TDB has a bulk loader - it loads empty databases.  It's idea of a batch 
is the whole data stream to be loaded.  It does not need to know the end 
of the batch when it starts a batch.  It's idea of a batch is millions 
of triples, beyond what can be buffered in RAM.

Bulk upload is not transactional.  Typically, it's a separate step using 
one of the bulk loader utilities. It handles triples and quads.  It does 
not use the BulkUpdateHandler.

For TDB, batches into a non-empty database are not special - they could 
be and it might be advantageous for some situations but currently it 
does not do anything special in this case.  In the future, Lizard (which 
is derived from TDB) would like to see batches for insertion; it could 
make use of bulk insertion even on non-empty databases.

TDB does nothing special about deletes (it does have, separately, have 
an optimized path for Graph.remove and Graph.clear).

When active, the bulk loader assumes total control over the database - 
any other operation (e.g. looking at the data) is likely to go wrong 
(very, very wrong!) - and it manipulate database details at a very low 
level.

For 100's of millions of triple, bulkloading is the only way to go.

== Towards Requirements

So we have related-but-different mechanisms in different places.

* Is bulk deletion an issue worth addressing?

Do any other systems have any bulk optimizations for deletion (but not 
"delete all").

* What about a mixture of adds/deletes?

* What is the contract e.g. parallel uses of Graph.find?

* What's the unit? Graphs, datasets other ?

Separately, there is a graph operation Graph.remove(S,P,O) where S,P,O 
can be Node.ANY so it's a pattern.

	Andy