You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Ewa Szwed <ew...@gmail.com> on 2014/06/17 10:17:44 UTC

tdbupdate vs tdbloader

Hi,
I have a question about performance of tdbupdate tool.
I would like to update my Jena TDB store that hosts freebase data with the
delta that I calculated between 2 consecutive freebase dumps.
I have 11 000 000 triples for deletion and 22 000 000 triples for
insertions.
My approach is that I divide these big sets into smaller batches - and use
tdbupdate with files passed as param for sparql deletes and sparql inserts.
I would expect that the performance is comparable to tdbloader but this is
not the case? Can this be improved? Can I used tdbloader instead of
tdbupdate for inserts update?
I appreciate any comment?

Re: tdbupdate vs tdbloader

Posted by Andy Seaborne <an...@apache.org>.
On 25/06/14 10:58, Ewa Szwed wrote:
> Hi, I have another question.
>
> During my process the tdbupdate for inserts (INSERT DATA) complains about
> the following triple:
> <http://rdf.freebase.com/ns/m.01065_86> <
> http://rdf.freebase.com/ns/common.topic.topic_equivalent_webpage> <
> http://www.espa
> %F1aescultura.es/es/obras_de_excelencia/museo_sorolla/la_hora_del_bano_valencia.html>
> .
>
> com.hp.hpl.jena.query.QueryParseException: Line 48497, column 113:
> org.apache.jena.iri.impl.IRIImplException:
> <http://www.espa%F1aescultura.es/es/obras_de_excelencia/museo_sorolla/la_hora_del_bano_valencia.html>
> Code: 28/NOT_DNS_NAME in HOST: The host component did not
> meet the restrictions on DNS names.

It could be made a warning in SPARQL as well but ideally, a legal URi 
would be better. %-encoding does not apply in host names; there is a 
different encoding scheme called "punycode" that then matches the DNS 
rules for DNS hostnames.  The whole IRI is made messy by all the 
different variations for the variable parts.

> For this case the validator (riot) issues just warning and tdbloaded
> inserted this triples without a problem.
> Is this possible to have this triple added using tdbupdate using some
> switches?

One radical solution is to use ñ not %F1 (i.e. decode) because it is 
legal in IRIs.


There is a high-risk way -

com.hp.hpl.jena.n3.IRIResolver.suppressExceptions();

but it stops all reporting.

Other workarounds are messy - what you need to do is change the resolver 
in the update request:

com.hp.hpl.jena.n3.IRIResolver resolverSilencer = new IRIResolver2() ;
UpdateRequest up = UpdateFactory.create() ;
up.setResolver(resolverSilencer) ;
UpdateFactory.parse(up, ....) ;


public static class IRIResolver2
      extends com.hp.hpl.jena.n3.IRIResolver {
         @Override
         public String resolve(String relURI) {
             if ( relURI.startsWith("http://") )
                 return relURI ;
             return super.resolve(relURI) ;
         }
     }

I have at least added

com.hp.hpl.jena.n3.IRIResolver.setShowExceptions(boolean) ;

so it can be both unset and reset.

> I would like to progress with this triple as well with tdbupdate to have
> consistent behaviour.

agreed - JENA-730 records the issue.

> Ewa


Re: tdbupdate vs tdbloader

Posted by Ewa Szwed <ew...@gmail.com>.
Hi, I have another question.

During my process the tdbupdate for inserts (INSERT DATA) complains about
the following triple:
<http://rdf.freebase.com/ns/m.01065_86> <
http://rdf.freebase.com/ns/common.topic.topic_equivalent_webpage> <
http://www.espa
%F1aescultura.es/es/obras_de_excelencia/museo_sorolla/la_hora_del_bano_valencia.html>
.

com.hp.hpl.jena.query.QueryParseException: Line 48497, column 113:
org.apache.jena.iri.impl.IRIImplException:
<http://www.espa%F1aescultura.es/es/obras_de_excelencia/museo_sorolla/la_hora_del_bano_valencia.html>
Code: 28/NOT_DNS_NAME in HOST: The host component did not
meet the restrictions on DNS names.
at
com.hp.hpl.jena.sparql.lang.ParserBase.throwParseException(ParserBase.java:661)
at com.hp.hpl.jena.sparql.lang.ParserBase.resolveIRI(ParserBase.java:255)
at
com.hp.hpl.jena.sparql.lang.ParserBase.resolveQuotedIRI(ParserBase.java:240)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.IRIREF(ARQParser.java:5417)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.iri(ARQParser.java:5359)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.GraphTerm(ARQParser.java:3819)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.VarOrTerm(ARQParser.java:3761)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.GraphNode(ARQParser.java:3675)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.Object(ARQParser.java:2929)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.ObjectList(ARQParser.java:2910)
at
com.hp.hpl.jena.sparql.lang.arq.ARQParser.PropertyListNotEmpty(ARQParser.java:2856)
at
com.hp.hpl.jena.sparql.lang.arq.ARQParser.TriplesSameSubject(ARQParser.java:2821)
at
com.hp.hpl.jena.sparql.lang.arq.ARQParser.TriplesTemplate(ARQParser.java:1968)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.Quads(ARQParser.java:1856)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.QuadData(ARQParser.java:1825)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.InsertData(ARQParser.java:1627)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.Update1(ARQParser.java:1467)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.Update(ARQParser.java:1405)
at com.hp.hpl.jena.sparql.lang.arq.ARQParser.UpdateUnit(ARQParser.java:71)
at
com.hp.hpl.jena.sparql.lang.ParserARQUpdate._parse(ParserARQUpdate.java:63)
at
com.hp.hpl.jena.sparql.lang.ParserARQUpdate.parse$(ParserARQUpdate.java:46)
at com.hp.hpl.jena.sparql.lang.UpdateParser.parse(UpdateParser.java:52)
at com.hp.hpl.jena.update.UpdateFactory.make(UpdateFactory.java:278)
at com.hp.hpl.jena.update.UpdateFactory.read(UpdateFactory.java:267)
at com.hp.hpl.jena.update.UpdateFactory.read(UpdateFactory.java:182)
at com.hp.hpl.jena.update.UpdateFactory.read(UpdateFactory.java:167)
at com.hp.hpl.jena.update.UpdateFactory.read(UpdateFactory.java:155)
at arq.update.execOneFile(update.java:114)
at arq.update.execUpdate(update.java:87)
at arq.cmdline.CmdUpdate.exec(CmdUpdate.java:58)
at arq.cmdline.CmdMain.mainMethod(CmdMain.java:101)
at arq.cmdline.CmdMain.mainRun(CmdMain.java:63)
at arq.cmdline.CmdMain.mainRun(CmdMain.java:50)
at tdb.tdbupdate.main(tdbupdate.java:36)

For this case the validator (riot) issues just warning and tdbloaded
inserted this triples without a problem.
Is this possible to have this triple added using tdbupdate using some
switches?
I would like to progress with this triple as well with tdbupdate to have
consistent behaviour.
Ewa


2014-06-25 10:43 GMT+01:00 Andy Seaborne <an...@apache.org>:

> On 24/06/14 22:56, Ewa Szwed wrote:
>
>> Hi,
>> DELETE DATA solved my problem - the performance is very good, I am doing a
>> test to delete in batches of 50 000 triples and it takes 20-30 second to
>> delete one batch! My JVM has -Xmx30g, I will try to increase the batch
>> size
>> even more. Thank you Stephen!
>>
>
> I'm glad you've fixed the immediate problem. DELETE DATA is right way (and
> faster).
>
> That said, DELETE WHERE and as few as 8K triples should not blow up the
> stack.
>
> JENA-729 records this report and has a reproducible test case.  It
> overflows the stack without TDB.
>
>         Andy
>

Re: tdbupdate vs tdbloader

Posted by Andy Seaborne <an...@apache.org>.
On 24/06/14 22:56, Ewa Szwed wrote:
> Hi,
> DELETE DATA solved my problem - the performance is very good, I am doing a
> test to delete in batches of 50 000 triples and it takes 20-30 second to
> delete one batch! My JVM has -Xmx30g, I will try to increase the batch size
> even more. Thank you Stephen!

I'm glad you've fixed the immediate problem. DELETE DATA is right way 
(and faster).

That said, DELETE WHERE and as few as 8K triples should not blow up the 
stack.

JENA-729 records this report and has a reproducible test case.  It 
overflows the stack without TDB.

	Andy

Re: tdbupdate vs tdbloader

Posted by Ewa Szwed <ew...@gmail.com>.
Hi,
DELETE DATA solved my problem - the performance is very good, I am doing a
test to delete in batches of 50 000 triples and it takes 20-30 second to
delete one batch! My JVM has -Xmx30g, I will try to increase the batch size
even more. Thank you Stephen!


2014-06-24 22:29 GMT+01:00 Stephen Allen <sa...@apache.org>:

> Also, use INSERT DATA as well for the insertion if you can.
>
>
> On Tue, Jun 24, 2014 at 5:21 PM, Stephen Allen <sa...@apache.org> wrote:
>
> > Hi Ewa,
> >
> > Try using the DELETE DATA query update form instead of DELETE WHERE.  The
> > DELETE DATA form is optimized for streaming and can handle many more
> > triples than the WHERE form.  The only limit on the number of triples to
> be
> > deleted is based on the amount of memory used in the heap by TDB's
> > uncommitted blocks, and not based on any limit in the SPARQL parser.
> >
> > -Stephen
> >
> >
> > On Tue, Jun 24, 2014 at 11:42 AM, Ewa Szwed <ew...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> More about my experiment case:
> >>
> >> My SPARQL delete file has 8000 triples:
> >>
> >> [root@freebase-m03 freebase_store_updates]# wc -l temp-file
> >> 8002 temp-file
> >>
> >> temp-file Content sample:
> >>
> >> DELETE WHERE {
> >> <http://rdf.freebase.com/ns/g.126tg6__p> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6__p> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6__r> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6__r> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6__r> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_fc> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_fc> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_fc> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_g0> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es
> .
> >> <http://rdf.freebase.com/ns/g.126tg6_g0> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name>
> >> "Συγγραφέας"@el
> >> .
> >> <http://rdf.freebase.com/ns/g.126tg6_g3> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.0np9r> .
> >> <http://rdf.freebase.com/ns/g.126tg6_g3> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice
> >> actor"@en
> >> .
> >> <http://rdf.freebase.com/ns/g.126tg6_g3> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.0np9r> .
> >> <http://rdf.freebase.com/ns/g.126tg6_gd> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_gd> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_gd> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_gj> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_gj> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_gj> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_js> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_js> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_js> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_kf> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_kf> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_kf> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_m6> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_m6> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_m6> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_m8> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_m8> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_m8> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mb> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mb> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_mb> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mf> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mf> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_mf> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_ml> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_ml> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_ml> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mn> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mn> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_mn> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mt> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mt> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >> <http://rdf.freebase.com/ns/g.126tg6_mt> <
> >> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mz> <
> >> http://rdf.freebase.com/ns/common.notable_for.object> <
> >> http://rdf.freebase.com/ns/m.09jd9nh> .
> >> <http://rdf.freebase.com/ns/g.126tg6_mz> <
> >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> >> Version"@en .
> >>
> >>
> >> Problematic command:
> >> $TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR
> >> --update=$FS_UPDATER_HOME
> >> /temp-file
> >>
> >> Error trace:
> >>
> >> 11:28:19 WARN  TDB                  :: Transaction not commited or
> >> aborted:
> >> Transaction: 1 : Mode=WRITE : State=ACTIVE :
> /data/servers/freebase_data/
> >>
> >> Exception in thread "main" java.lang.StackOverflowError
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >> at
> >>
> >>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> >> at
> >>
> >>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> >>
> >> Software version:
> >> more ReleaseNotes-Jena.txt
> >> Release Notes
> >> =============
> >>
> >> ==== Jena 2.11.0
> >>
> >>
> >> Please note that when my temp-file was 5000 triples or smaller it worked
> >> just fine.
> >>
> >> Please advise because I have a feeling I could increase the size to 10
> 000
> >> - 20 000 triples with performance improvement.
> >>
> >> When batch was:
> >>
> >> 500 triples, it took 20 seconds fro batch
> >>
> >> when
> >>
> >> 5000 triples it took 30 seconds so I expect good results for even bigger
> >> batches but this StackOverFlow is a blocker.
> >>
> >> Best Regards,
> >>
> >> Ewa
> >>
> >>
> >> 2014-06-24 14:33 GMT+01:00 Andy Seaborne <an...@apache.org>:
> >>
> >> > On 23/06/14 18:27, Ewa Szwed wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I migrated my Freebase Jena TDB setup to SSD host and continued my
> >> >> tdbupdate deletes experiments. The expectation was the transaction
> >> times
> >> >> are improved. I was able to see the improvements:
> >> >>
> >> >> batch 5000 triples - 1 min 20 sec / batch
> >> >>
> >> >> Now when I reconfigure my bath size to 8000 to see if the time stays
> at
> >> >> this level for single batch I see:
> >> >>
> >> >>
> >> >> WARN TDB:: Transaction not commited or aborted: Transaction: 1 :
> >> >> Mode=WRITE
> >> >> : State=ACTIVE
> >> >>   : /data/servers/freebase_data/
> >> >>
> >> >> Exception in thread "main" java.lang.StackOverflowError
> >> >> at
> >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
> >> >> hasNext(RepeatApplyIterator.java:42)
> >> >> at
> >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
> >> >> hasNext(SolverLib.java:197)
> >> >> at
> >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
> >> >> hasNext(RepeatApplyIterator.java:46)
> >> >> at
> >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
> >> >> hasNext(SolverLib.java:197)
> >> >> at
> >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
> >> >> hasNext(RepeatApplyIterator.java:46)
> >> >>
> >> >> Is this a bug?
> >> >> Please advise how should I proceed?
> >> >>
> >> >
> >> > I'd need to know more about what's going to say but it isn't anything
> I
> >> > recognize.
> >> >
> >> > Could you provide the whole stack trace (well, the start and finish of
> >> the
> >> > boring bit :-) and details of the update being attempted. And could
> you
> >> > please confirm the version you're using?
> >> >
> >> >         Thanks
> >> >         Andy
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>

Re: tdbupdate vs tdbloader

Posted by Stephen Allen <sa...@apache.org>.
Also, use INSERT DATA as well for the insertion if you can.


On Tue, Jun 24, 2014 at 5:21 PM, Stephen Allen <sa...@apache.org> wrote:

> Hi Ewa,
>
> Try using the DELETE DATA query update form instead of DELETE WHERE.  The
> DELETE DATA form is optimized for streaming and can handle many more
> triples than the WHERE form.  The only limit on the number of triples to be
> deleted is based on the amount of memory used in the heap by TDB's
> uncommitted blocks, and not based on any limit in the SPARQL parser.
>
> -Stephen
>
>
> On Tue, Jun 24, 2014 at 11:42 AM, Ewa Szwed <ew...@gmail.com> wrote:
>
>> Hi,
>>
>> More about my experiment case:
>>
>> My SPARQL delete file has 8000 triples:
>>
>> [root@freebase-m03 freebase_store_updates]# wc -l temp-file
>> 8002 temp-file
>>
>> temp-file Content sample:
>>
>> DELETE WHERE {
>> <http://rdf.freebase.com/ns/g.126tg6__p> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6__p> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6__r> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6__r> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6__r> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_fc> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_fc> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_fc> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_g0> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es .
>> <http://rdf.freebase.com/ns/g.126tg6_g0> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name>
>> "Συγγραφέας"@el
>> .
>> <http://rdf.freebase.com/ns/g.126tg6_g3> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.0np9r> .
>> <http://rdf.freebase.com/ns/g.126tg6_g3> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice
>> actor"@en
>> .
>> <http://rdf.freebase.com/ns/g.126tg6_g3> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.0np9r> .
>> <http://rdf.freebase.com/ns/g.126tg6_gd> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_gd> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_gd> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_gj> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_gj> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_gj> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_js> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_js> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_js> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_kf> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_kf> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_kf> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_m6> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_m6> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_m6> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_m8> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_m8> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_m8> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mb> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mb> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_mb> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mf> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mf> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_mf> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_ml> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_ml> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_ml> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mn> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mn> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_mn> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mt> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mt> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>> <http://rdf.freebase.com/ns/g.126tg6_mt> <
>> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mz> <
>> http://rdf.freebase.com/ns/common.notable_for.object> <
>> http://rdf.freebase.com/ns/m.09jd9nh> .
>> <http://rdf.freebase.com/ns/g.126tg6_mz> <
>> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
>> Version"@en .
>>
>>
>> Problematic command:
>> $TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR
>> --update=$FS_UPDATER_HOME
>> /temp-file
>>
>> Error trace:
>>
>> 11:28:19 WARN  TDB                  :: Transaction not commited or
>> aborted:
>> Transaction: 1 : Mode=WRITE : State=ACTIVE : /data/servers/freebase_data/
>>
>> Exception in thread "main" java.lang.StackOverflowError
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>> at
>>
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>> at
>>
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>>
>> Software version:
>> more ReleaseNotes-Jena.txt
>> Release Notes
>> =============
>>
>> ==== Jena 2.11.0
>>
>>
>> Please note that when my temp-file was 5000 triples or smaller it worked
>> just fine.
>>
>> Please advise because I have a feeling I could increase the size to 10 000
>> - 20 000 triples with performance improvement.
>>
>> When batch was:
>>
>> 500 triples, it took 20 seconds fro batch
>>
>> when
>>
>> 5000 triples it took 30 seconds so I expect good results for even bigger
>> batches but this StackOverFlow is a blocker.
>>
>> Best Regards,
>>
>> Ewa
>>
>>
>> 2014-06-24 14:33 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>
>> > On 23/06/14 18:27, Ewa Szwed wrote:
>> >
>> >> Hi,
>> >>
>> >> I migrated my Freebase Jena TDB setup to SSD host and continued my
>> >> tdbupdate deletes experiments. The expectation was the transaction
>> times
>> >> are improved. I was able to see the improvements:
>> >>
>> >> batch 5000 triples - 1 min 20 sec / batch
>> >>
>> >> Now when I reconfigure my bath size to 8000 to see if the time stays at
>> >> this level for single batch I see:
>> >>
>> >>
>> >> WARN TDB:: Transaction not commited or aborted: Transaction: 1 :
>> >> Mode=WRITE
>> >> : State=ACTIVE
>> >>   : /data/servers/freebase_data/
>> >>
>> >> Exception in thread "main" java.lang.StackOverflowError
>> >> at
>> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
>> >> hasNext(RepeatApplyIterator.java:42)
>> >> at
>> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
>> >> hasNext(SolverLib.java:197)
>> >> at
>> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
>> >> hasNext(RepeatApplyIterator.java:46)
>> >> at
>> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
>> >> hasNext(SolverLib.java:197)
>> >> at
>> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
>> >> hasNext(RepeatApplyIterator.java:46)
>> >>
>> >> Is this a bug?
>> >> Please advise how should I proceed?
>> >>
>> >
>> > I'd need to know more about what's going to say but it isn't anything I
>> > recognize.
>> >
>> > Could you provide the whole stack trace (well, the start and finish of
>> the
>> > boring bit :-) and details of the update being attempted. And could you
>> > please confirm the version you're using?
>> >
>> >         Thanks
>> >         Andy
>> >
>> >
>> >
>> >
>>
>
>

Re: tdbupdate vs tdbloader

Posted by Stephen Allen <sa...@apache.org>.
Hi Ewa,

Try using the DELETE DATA query update form instead of DELETE WHERE.  The
DELETE DATA form is optimized for streaming and can handle many more
triples than the WHERE form.  The only limit on the number of triples to be
deleted is based on the amount of memory used in the heap by TDB's
uncommitted blocks, and not based on any limit in the SPARQL parser.

-Stephen


On Tue, Jun 24, 2014 at 11:42 AM, Ewa Szwed <ew...@gmail.com> wrote:

> Hi,
>
> More about my experiment case:
>
> My SPARQL delete file has 8000 triples:
>
> [root@freebase-m03 freebase_store_updates]# wc -l temp-file
> 8002 temp-file
>
> temp-file Content sample:
>
> DELETE WHERE {
> <http://rdf.freebase.com/ns/g.126tg6__p> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6__p> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6__r> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6__r> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6__r> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_fc> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_fc> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_fc> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_g0> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es .
> <http://rdf.freebase.com/ns/g.126tg6_g0> <
> http://rdf.freebase.com/ns/common.notable_for.display_name>
> "Συγγραφέας"@el
> .
> <http://rdf.freebase.com/ns/g.126tg6_g3> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.0np9r> .
> <http://rdf.freebase.com/ns/g.126tg6_g3> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice
> actor"@en
> .
> <http://rdf.freebase.com/ns/g.126tg6_g3> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.0np9r> .
> <http://rdf.freebase.com/ns/g.126tg6_gd> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_gd> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_gd> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_gj> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_gj> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_gj> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_js> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_js> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_js> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_kf> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_kf> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_kf> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_m6> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_m6> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_m6> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_m8> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_m8> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_m8> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mb> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mb> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_mb> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mf> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mf> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_mf> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_ml> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_ml> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_ml> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mn> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mn> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_mn> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mt> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mt> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
> <http://rdf.freebase.com/ns/g.126tg6_mt> <
> http://rdf.freebase.com/ns/common.notable_for.notable_object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mz> <
> http://rdf.freebase.com/ns/common.notable_for.object> <
> http://rdf.freebase.com/ns/m.09jd9nh> .
> <http://rdf.freebase.com/ns/g.126tg6_mz> <
> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
> Version"@en .
>
>
> Problematic command:
> $TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR
> --update=$FS_UPDATER_HOME
> /temp-file
>
> Error trace:
>
> 11:28:19 WARN  TDB                  :: Transaction not commited or aborted:
> Transaction: 1 : Mode=WRITE : State=ACTIVE : /data/servers/freebase_data/
>
> Exception in thread "main" java.lang.StackOverflowError
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
>
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
>
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
>
> Software version:
> more ReleaseNotes-Jena.txt
> Release Notes
> =============
>
> ==== Jena 2.11.0
>
>
> Please note that when my temp-file was 5000 triples or smaller it worked
> just fine.
>
> Please advise because I have a feeling I could increase the size to 10 000
> - 20 000 triples with performance improvement.
>
> When batch was:
>
> 500 triples, it took 20 seconds fro batch
>
> when
>
> 5000 triples it took 30 seconds so I expect good results for even bigger
> batches but this StackOverFlow is a blocker.
>
> Best Regards,
>
> Ewa
>
>
> 2014-06-24 14:33 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
> > On 23/06/14 18:27, Ewa Szwed wrote:
> >
> >> Hi,
> >>
> >> I migrated my Freebase Jena TDB setup to SSD host and continued my
> >> tdbupdate deletes experiments. The expectation was the transaction times
> >> are improved. I was able to see the improvements:
> >>
> >> batch 5000 triples - 1 min 20 sec / batch
> >>
> >> Now when I reconfigure my bath size to 8000 to see if the time stays at
> >> this level for single batch I see:
> >>
> >>
> >> WARN TDB:: Transaction not commited or aborted: Transaction: 1 :
> >> Mode=WRITE
> >> : State=ACTIVE
> >>   : /data/servers/freebase_data/
> >>
> >> Exception in thread "main" java.lang.StackOverflowError
> >> at
> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
> >> hasNext(RepeatApplyIterator.java:42)
> >> at
> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
> >> hasNext(SolverLib.java:197)
> >> at
> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
> >> hasNext(RepeatApplyIterator.java:46)
> >> at
> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
> >> hasNext(SolverLib.java:197)
> >> at
> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.
> >> hasNext(RepeatApplyIterator.java:46)
> >>
> >> Is this a bug?
> >> Please advise how should I proceed?
> >>
> >
> > I'd need to know more about what's going to say but it isn't anything I
> > recognize.
> >
> > Could you provide the whole stack trace (well, the start and finish of
> the
> > boring bit :-) and details of the update being attempted. And could you
> > please confirm the version you're using?
> >
> >         Thanks
> >         Andy
> >
> >
> >
> >
>

Re: tdbupdate vs tdbloader

Posted by Ewa Szwed <ew...@gmail.com>.
Hi,

More about my experiment case:

My SPARQL delete file has 8000 triples:

[root@freebase-m03 freebase_store_updates]# wc -l temp-file
8002 temp-file

temp-file Content sample:

DELETE WHERE {
<http://rdf.freebase.com/ns/g.126tg6__p> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6__p> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6__r> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6__r> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6__r> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_fc> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_fc> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_fc> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_g0> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es .
<http://rdf.freebase.com/ns/g.126tg6_g0> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Συγγραφέας"@el
.
<http://rdf.freebase.com/ns/g.126tg6_g3> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.0np9r> .
<http://rdf.freebase.com/ns/g.126tg6_g3> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice actor"@en
.
<http://rdf.freebase.com/ns/g.126tg6_g3> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.0np9r> .
<http://rdf.freebase.com/ns/g.126tg6_gd> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_gd> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_gd> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_gj> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_gj> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_gj> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_js> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_js> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_js> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_kf> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_kf> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_kf> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_m6> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_m6> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_m6> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_m8> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_m8> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_m8> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mb> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mb> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_mb> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mf> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mf> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_mf> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_ml> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_ml> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_ml> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mn> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mn> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_mn> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mt> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mt> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .
<http://rdf.freebase.com/ns/g.126tg6_mt> <
http://rdf.freebase.com/ns/common.notable_for.notable_object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mz> <
http://rdf.freebase.com/ns/common.notable_for.object> <
http://rdf.freebase.com/ns/m.09jd9nh> .
<http://rdf.freebase.com/ns/g.126tg6_mz> <
http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical
Version"@en .


Problematic command:
$TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR --update=$FS_UPDATER_HOME
/temp-file

Error trace:

11:28:19 WARN  TDB                  :: Transaction not commited or aborted:
Transaction: 1 : Mode=WRITE : State=ACTIVE : /data/servers/freebase_data/

Exception in thread "main" java.lang.StackOverflowError
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)

Software version:
more ReleaseNotes-Jena.txt
Release Notes
=============

==== Jena 2.11.0


Please note that when my temp-file was 5000 triples or smaller it worked
just fine.

Please advise because I have a feeling I could increase the size to 10 000
- 20 000 triples with performance improvement.

When batch was:

500 triples, it took 20 seconds fro batch

when

5000 triples it took 30 seconds so I expect good results for even bigger
batches but this StackOverFlow is a blocker.

Best Regards,

Ewa


2014-06-24 14:33 GMT+01:00 Andy Seaborne <an...@apache.org>:

> On 23/06/14 18:27, Ewa Szwed wrote:
>
>> Hi,
>>
>> I migrated my Freebase Jena TDB setup to SSD host and continued my
>> tdbupdate deletes experiments. The expectation was the transaction times
>> are improved. I was able to see the improvements:
>>
>> batch 5000 triples - 1 min 20 sec / batch
>>
>> Now when I reconfigure my bath size to 8000 to see if the time stays at
>> this level for single batch I see:
>>
>>
>> WARN TDB:: Transaction not commited or aborted: Transaction: 1 :
>> Mode=WRITE
>> : State=ACTIVE
>>   : /data/servers/freebase_data/
>>
>> Exception in thread "main" java.lang.StackOverflowError
>> at
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.
>> hasNext(RepeatApplyIterator.java:42)
>> at
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
>> hasNext(SolverLib.java:197)
>> at
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.
>> hasNext(RepeatApplyIterator.java:46)
>> at
>> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.
>> hasNext(SolverLib.java:197)
>> at
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.
>> hasNext(RepeatApplyIterator.java:46)
>>
>> Is this a bug?
>> Please advise how should I proceed?
>>
>
> I'd need to know more about what's going to say but it isn't anything I
> recognize.
>
> Could you provide the whole stack trace (well, the start and finish of the
> boring bit :-) and details of the update being attempted. And could you
> please confirm the version you're using?
>
>         Thanks
>         Andy
>
>
>
>

Re: tdbupdate vs tdbloader

Posted by Andy Seaborne <an...@apache.org>.
On 23/06/14 18:27, Ewa Szwed wrote:
> Hi,
> I migrated my Freebase Jena TDB setup to SSD host and continued my
> tdbupdate deletes experiments. The expectation was the transaction times
> are improved. I was able to see the improvements:
>
> batch 5000 triples - 1 min 20 sec / batch
>
> Now when I reconfigure my bath size to 8000 to see if the time stays at
> this level for single batch I see:
>
>
> WARN TDB:: Transaction not commited or aborted: Transaction: 1 : Mode=WRITE
> : State=ACTIVE
>   : /data/servers/freebase_data/
>
> Exception in thread "main" java.lang.StackOverflowError
> at
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:42)
> at
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
> at
> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
> at
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
>
> Is this a bug?
> Please advise how should I proceed?

I'd need to know more about what's going to say but it isn't anything I 
recognize.

Could you provide the whole stack trace (well, the start and finish of 
the boring bit :-) and details of the update being attempted. And could 
you please confirm the version you're using?

	Thanks
	Andy




Re: tdbupdate vs tdbloader

Posted by Ewa Szwed <ew...@gmail.com>.
Hi,
I migrated my Freebase Jena TDB setup to SSD host and continued my
tdbupdate deletes experiments. The expectation was the transaction times
are improved. I was able to see the improvements:

batch 5000 triples - 1 min 20 sec / batch

Now when I reconfigure my bath size to 8000 to see if the time stays at
this level for single batch I see:


WARN TDB:: Transaction not commited or aborted: Transaction: 1 : Mode=WRITE
: State=ACTIVE
 : /data/servers/freebase_data/

Exception in thread "main" java.lang.StackOverflowError
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:42)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)
at
com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46)

Is this a bug?
Please advise how should I proceed?


2014-06-18 14:40 GMT+01:00 Ewa Szwed <ew...@gmail.com>:

> Hi,
> Is that expected that tdbupdate deletes take much longer than inserts.
> For inserts I have divided my set into batches of 500 000 triples and I
> insert 1 batch in 5 minutes; the deletes: I created a batch of 100 000
> triples and it has been running whole night and is still running? Can this
> be tuned? Now I am experimenting with small batches for deletes.
> Here are my results so far:
> batch 1000 triples - 2mins/batch
> batch 2000 triples - 2,5 mins/batch
> batch 4000 triples - 6 mins/batch
> I have enabled gc log and I see gc runs very often.
> Can JVM tuning improve performance of tdbupdate deletes much?
>
>
> 2014-06-17 15:37 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
> On 17/06/14 09:17, Ewa Szwed wrote:
>>
>>> Hi,
>>> I have a question about performance of tdbupdate tool.
>>> I would like to update my Jena TDB store that hosts freebase data with
>>> the
>>> delta that I calculated between 2 consecutive freebase dumps.
>>> I have 11 000 000 triples for deletion and 22 000 000 triples for
>>> insertions.
>>> My approach is that I divide these big sets into smaller batches - and
>>> use
>>> tdbupdate with files passed as param for sparql deletes and sparql
>>> inserts.
>>> I would expect that the performance is comparable to tdbloader but this
>>> is
>>> not the case? Can this be improved? Can I used tdbloader instead of
>>> tdbupdate for inserts update?
>>> I appreciate any comment?
>>>
>>>
>> tdbloader only makes a difference when loading an empty dataset -
>> tdbloader knows how to manipulate the indexes in a better fashion than
>> simple letting the indexes incrementally update and tdbloader2 knows how to
>> build the index b+trees directly.
>>
>> tdbupdate is applying SPARQL updates and does not exploit the
>> implementation of TDB.
>>
>> Normally, I'd suggest considering working with N-Quads and doing insert
>> and delete on those (if no bnodes) but the reload size of Freebase makes
>> that a bit of a nuisance.
>>
>> With 11e6 and 22e6, batching in smaller groups is going to be a good idea
>> otherwise the transaction mechanism will (there is spill-to-disk (see TDB.
>> transactionJournalWriteBlockMode) but it will be slow because there is a
>> disk.
>>
>> Having the database on an SSD will make a big difference  It greatly
>> speeds up transactions.
>>
>>                 Andy
>>
>>
>

Re: tdbupdate vs tdbloader

Posted by Ewa Szwed <ew...@gmail.com>.
Hi,
Is that expected that tdbupdate deletes take much longer than inserts.
For inserts I have divided my set into batches of 500 000 triples and I
insert 1 batch in 5 minutes; the deletes: I created a batch of 100 000
triples and it has been running whole night and is still running? Can this
be tuned? Now I am experimenting with small batches for deletes.
Here are my results so far:
batch 1000 triples - 2mins/batch
batch 2000 triples - 2,5 mins/batch
batch 4000 triples - 6 mins/batch
I have enabled gc log and I see gc runs very often.
Can JVM tuning improve performance of tdbupdate deletes much?


2014-06-17 15:37 GMT+01:00 Andy Seaborne <an...@apache.org>:

> On 17/06/14 09:17, Ewa Szwed wrote:
>
>> Hi,
>> I have a question about performance of tdbupdate tool.
>> I would like to update my Jena TDB store that hosts freebase data with the
>> delta that I calculated between 2 consecutive freebase dumps.
>> I have 11 000 000 triples for deletion and 22 000 000 triples for
>> insertions.
>> My approach is that I divide these big sets into smaller batches - and use
>> tdbupdate with files passed as param for sparql deletes and sparql
>> inserts.
>> I would expect that the performance is comparable to tdbloader but this is
>> not the case? Can this be improved? Can I used tdbloader instead of
>> tdbupdate for inserts update?
>> I appreciate any comment?
>>
>>
> tdbloader only makes a difference when loading an empty dataset -
> tdbloader knows how to manipulate the indexes in a better fashion than
> simple letting the indexes incrementally update and tdbloader2 knows how to
> build the index b+trees directly.
>
> tdbupdate is applying SPARQL updates and does not exploit the
> implementation of TDB.
>
> Normally, I'd suggest considering working with N-Quads and doing insert
> and delete on those (if no bnodes) but the reload size of Freebase makes
> that a bit of a nuisance.
>
> With 11e6 and 22e6, batching in smaller groups is going to be a good idea
> otherwise the transaction mechanism will (there is spill-to-disk (see TDB.
> transactionJournalWriteBlockMode) but it will be slow because there is a
> disk.
>
> Having the database on an SSD will make a big difference  It greatly
> speeds up transactions.
>
>                 Andy
>
>

Re: tdbupdate vs tdbloader

Posted by Andy Seaborne <an...@apache.org>.
On 17/06/14 09:17, Ewa Szwed wrote:
> Hi,
> I have a question about performance of tdbupdate tool.
> I would like to update my Jena TDB store that hosts freebase data with the
> delta that I calculated between 2 consecutive freebase dumps.
> I have 11 000 000 triples for deletion and 22 000 000 triples for
> insertions.
> My approach is that I divide these big sets into smaller batches - and use
> tdbupdate with files passed as param for sparql deletes and sparql inserts.
> I would expect that the performance is comparable to tdbloader but this is
> not the case? Can this be improved? Can I used tdbloader instead of
> tdbupdate for inserts update?
> I appreciate any comment?
>

tdbloader only makes a difference when loading an empty dataset - 
tdbloader knows how to manipulate the indexes in a better fashion than 
simple letting the indexes incrementally update and tdbloader2 knows how 
to build the index b+trees directly.

tdbupdate is applying SPARQL updates and does not exploit the 
implementation of TDB.

Normally, I'd suggest considering working with N-Quads and doing insert 
and delete on those (if no bnodes) but the reload size of Freebase makes 
that a bit of a nuisance.

With 11e6 and 22e6, batching in smaller groups is going to be a good 
idea otherwise the transaction mechanism will (there is spill-to-disk 
(see TDB.transactionJournalWriteBlockMode) but it will be slow because 
there is a disk.

Having the database on an SSD will make a big difference  It greatly 
speeds up transactions.

		Andy