Fast & Reliable Node Parser

[Paul Houle <on...@gmail.com>]

     I've had many requests to port some of the advances in my
infovore framework to Jena and now I'm getting around to that.

     My program Infovore at github

https://github.com/paulhoule/infovore

     has a module called "parallel super eyeball" which,  like the
eyeball program,  checks a RDF file for trouble,  but does not crash
when it finds it.  One simplifing trick was to accept only N-Triples
and close variants,  such as the Freebase export files.  This means I
can reliable break up triples into nodes by breaking on the first two
bits of whitespace,  then parse the nodes separately.

     I hacked away at the triple parser in Jena to produce something
that parses a single node and I did it in a surgical way so there is a
pretty good chance it is correct.  The result is here:

https://github.com/paulhoule/infovore/tree/master/
millipede/src/main/java/com/ontology2/rdf/parser

     The real trouble with it is that it is terribly slow,  so slow
that I was about to give up on it before introducing a parse cache,
which is the function createNodeParseCache() in

https://github.com/paulhoule/infovore/blob/master/
millipede/src/main/java/com/ontology2/rdf/JenaUtil.java

     This sped it up to the point where I was not motivated to work
out how to speed it up,  but this work should happen.  I'm sure that
the parser is doing a lot of set-up work,  some of which is
superfluous,  and also I'm certain that a handwritten parser could be
faster than the generated parser as well.  Seeing how many billion of
triples there are ought there,  a handwritten node parser may be worth
the effort.

----

    On another note,  I couldn't help but notice that it's easy to
fill up memory with identical Node objects as seen in the following
test:

https://github.com/paulhoule/infovore/blob/master/
millipede/src/test/java/com/ontology2/rdf/UnderstandNodeMemoryBehavior.java

    Given that many graphs repeat the same node values a lot,  I wrote
some Economizer classes,  tested in there,  that make a cache for
recently created Node and Triple objects.  Perhaps I was being a bit
silly to expect to sort very large arrays of triples in memory,  but I
found I was greatly able to reduce the memory required by using
"Economization".

Any thoughts?