You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Jean-Marc Vanel <je...@gmail.com> on 2017/12/01 11:49:27 UTC

Slow Lucene query

Hi

The time for a SPARQL + Lucene query is more than 1 mn.
It used to be around one second, and then the database grew.
Here is a typical query run for a lookup service similar to dbpedia lookup:

PREFIX text: <http://jena.apache.org/text#>
PREFIX form: <
http://raw.githubusercontent.com/jmvanel/semantic_forms/master/vocabulary/forms.owl.ttl#
>
SELECT DISTINCT ?thing ?COUNT WHERE {
  graph ?g {
    ?thing text:query ( 'Jean-Marc' ) .
  }
  graph ?g1 {
    ?thing a ?CLASS .
  }
  OPTIONAL {
   graph ?grCount {
    ?thing form:linksCount ?COUNT.
  } }
} ORDER BY DESC(?COUNT)
LIMIT 10

Here is a simpler query that is also slow:

PREFIX text: <http://jena.apache.org/text#>
SELECT DISTINCT ?thing ?COUNT WHERE {
  graph ?g {
    ?thing text:query ( 'Jean-Marc' ) .
  }
} ORDER BY DESC(?COUNT)
LIMIT 10

Run it with:
time wget -O semantic-forms.cc_select-ui.txt
http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A+%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%0D%0ALIMIT+10

Or, if you want to use YasGUI , the endpoint is
http://semantic-forms.cc:9112/sparql

*Statistics on the database*
268 graphs and 588 864 triples.

# Count graphs and triples
SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
    WHERE {
  { GRAPH ?GR { } }
  UNION
  { GRAPH ?GR1 { ?s ?p ?o . } }
}

Result: 2 rows
"grc" "trc"
"268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
http://www.w3.org/2001/XMLSchema#integer

(I'm not sure this the right way to count, but it gives figures :) )

You can reproduce the query with this UI :

http://semantic-forms.cc:9112/select-ui?query=%23+Count+graphs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D

This is using Jena 3.5.0. with TDB 1 .
Here is a stack made with kill -3 when the app. was working hard;
I put in bold a suspect line.

"application-akka.actor.default-dispatcher-1351" #2683 prio=5 os_prio=0
tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
at
org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
at
org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldReader.java:91)
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:234)
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:445)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:112)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:77)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at
org.apache.jena.query.text.TextIndexLucene.query(TextIndexLucene.java:370)
at org.apache.jena.query.text.TextQueryPF.performQuery(TextQueryPF.java:290)
at
org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQueryPF.java:267)
at
org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189.call(Unknown
Source)
at
org.apache.jena.ext.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5065)
at
org.apache.jena.ext.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
at
org.apache.jena.ext.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
at
org.apache.jena.ext.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
* - locked <0x00000000ef9d34f8> (a
org.apache.jena.ext.com.google.common.cache.LocalCache$StrongAccessEntry)*
at
org.apache.jena.ext.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
at
org.apache.jena.ext.com.google.common.cache.LocalCache.get(LocalCache.java:4154)
at
org.apache.jena.ext.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5060)
at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGuava.java:58)
at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
at
org.apache.jena.query.text.TextQueryPF.variableSubject(TextQueryPF.java:227)
at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
at
org.apache.jena.sparql.pfunction.PropertyFunctionBase$RepeatApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
at
org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.makeNextStage(QueryIterRepeatApply.java:108)
at
org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:65)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterProcedure.hasNextBinding(QueryIterProcedure.java:73)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterProcessBinding.hasNextBinding(QueryIterProcessBinding.java:66)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$QueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:74)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
at
org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:45)
at
org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195)
at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
at
org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:53)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.makeNextStage(QueryIterRepeatApply.java:101)
at
org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:65)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasNextBinding(QueryIterConvert.java:58)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initializeIterator(QueryIterTopN.java:98)
at
org.apache.jena.atlas.iterator.IteratorDelayedInitialization.init(IteratorDelayedInitialization.java:40)
at
org.apache.jena.atlas.iterator.IteratorDelayedInitialization.hasNext(IteratorDelayedInitialization.java:50)
at
org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:53)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:74)
at
org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNext(ResultSetCheckCondition.java:55)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
at
scala.collection.TraversableOnce$class.toIterable(TraversableOnce.scala:296)
at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
at
deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparqlSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
at
deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparqlSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
at controllers.Application$.time(Application.scala:8)
at
deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSelectQueryVariablesNT(SPARQLHelpers.scala:370)
at
controllers.Application$.sparqlSelectQueryVariablesNT(Application.scala:8)
at
deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.apply(SPARQLHelpers.scala:361)
at
deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.apply(SPARQLHelpers.scala:361)
at
org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(JenaDatasetStore.scala:17)
at scala.util.Try$.apply(Try.scala:192)
at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
at
deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSelectQueryVariables(SPARQLHelpers.scala:359)
at controllers.Application$.sparqlSelectQueryVariables(Application.scala:8)
at
deductions.runtime.services.Lookup$class.searchStringOrClass(Lookup.scala:76)
at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
at controllers.Application$.lookup(Application.scala:8)
at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:199)
at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:193)
at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
at play.api.mvc.Action$.invokeBlock(Action.scala:498)
at play.api.mvc.Action$.invokeBlock(Action.scala:495)
at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
at
play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
at
play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
at
play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:111)
at
play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:110)
at scala.Option.map(Option.scala:146)
at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

-- 
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui

Re: Slow Lucene query

Posted by Jean-Marc Vanel <je...@gmail.com>.
Thanks Osma

Now everything is fine, querying is quick again, for instance
http://semantic-forms.cc:9112/search?q=Gandon
(I have loaded the official ISWC 1217 RDF file )


2017-12-08 8:09 GMT+01:00 Osma Suominen <os...@helsinki.fi>:

> Andy Seaborne kirjoitti 05.12.2017 klo 12:13:
>
>> I understand that I must deactivate "graph-specific indexing" that is not
>>> not needed, and slows the query.
>>> The doc. does not say so, but I think rebuilding the index is
>>> appropriate.
>>>
>>
>> Yes.
>>
>
> Well, in this case (deactivating graphField), not strictly necessary...
> The old graph values will still be there in the index but they won't be
> used during query time if graphField is not set, so there shouldn't be any
> slowdown due to this.
>
> But in general, it is a good idea to rebuild the text index after any
> change in the configuration.
>
> -Osma
>
> --
> Osma Suominen
> D.Sc. (Tech), Information Systems Specialist
> National Library of Finland
> P.O. Box 26 (Kaikukatu 4)
> 00014 HELSINGIN YLIOPISTO
> Tel. +358 50 3199529
> osma.suominen@helsinki.fi
> http://www.nationallibrary.fi
>



-- 
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui

Re: Slow Lucene query

Posted by Osma Suominen <os...@helsinki.fi>.
Andy Seaborne kirjoitti 05.12.2017 klo 12:13:
>> I understand that I must deactivate "graph-specific indexing" that is not
>> not needed, and slows the query.
>> The doc. does not say so, but I think rebuilding the index is 
>> appropriate.
> 
> Yes.

Well, in this case (deactivating graphField), not strictly necessary... 
The old graph values will still be there in the index but they won't be 
used during query time if graphField is not set, so there shouldn't be 
any slowdown due to this.

But in general, it is a good idea to rebuild the text index after any 
change in the configuration.

-Osma

-- 
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Kaikukatu 4)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.suominen@helsinki.fi
http://www.nationallibrary.fi

Re: Slow Lucene query

Posted by Andy Seaborne <an...@apache.org>.

On 05/12/17 08:20, Jean-Marc Vanel wrote:
> I understand that I must deactivate "graph-specific indexing" that is not
> not needed, and slows the query.
> The doc. does not say so, but I think rebuilding the index is appropriate.

Yes.

 > entMap.setGraphField("graph")

There is a specific field in the index for the graph so if you don't 
want it, the Lucene document format is changing.

     Andy

Re: Slow Lucene query

Posted by Jean-Marc Vanel <je...@gmail.com>.
Sorry Lorentz , yes ,
the query was badly pasted.
It is currently in the services:

PREFIX text: <http://jena.apache.org/text#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX form: <
http://raw.githubusercontent.com/jmvanel/semantic_forms/master/vocabulary/forms.owl.ttl#
>
SELECT DISTINCT ?thing ?COUNT WHERE {
*  ?thing text:query ( 'caractères' ) .        *
  graph ?g1 {
    ?thing a ?CLASS .
  }
  OPTIONAL {
   graph ?grCount {
    ?thing form:linksCount ?COUNT.
  } }
}
ORDER BY DESC(?COUNT)
LIMIT 10

I forgot to say that the triples
    ?thing form:linksCount ?COUNT.
are updated at every database change to achieve an ordering of the results
similar to Google page rank , based on RDF triples. It was mentioned some
months ago on this list as a possible implementation.


Now everything is fixed; done:

   - deactivate "graph-specific indexing" cf
   https://jena.apache.org/documentation/query/text-query.html#graph-specific-indexing
   - ?thing text:query $userString
   should not be inside GRAPH {} block
   - re-created the Lucene text index

Now /search runs in 0.3 s :
http://semantic-forms.cc:9112/search?q=S%C3%A9bastien

It should run quicker by pre-compiling the query.

The dbpedia lookup - like service runs around 0.2 s :
http://semantic-forms.cc:9112/lookup?QueryString=S%C3%A9bastien

2017-12-05 9:39 GMT+01:00 Lorenz Buehmann <
buehmann@informatik.uni-leipzig.de>:

> >> Here is a simple query that is slow:
> > PREFIX text: <http://jena.apache.org/text#>
> > SELECT DISTINCT ?thing ?COUNT WHERE {
> >     graph ?g {
> >       ?thing text:query ( 'Jean-Marc' ) .
> >     }
> > } ORDER BY DESC(?COUNT)
> > LIMIT 10
> >
> >
> >
> ?COUNT is not bound here
>
>
> On 05.12.2017 09:20, Jean-Marc Vanel wrote:
> > 2017-12-04 10:21 GMT+01:00 Andy Seaborne <an...@apache.org>:
> >
> >> Hi - your messages loose the indication of who wrote what.
> >>
> > Sorry, I edited all , putting "AS:" and "JMV:" prefixes,
> > and removing the less important stuff.
> >
> >
> >> AS:
> >>
> >
> >> Maybe it's this part of the query that causes some or all of the costs
> >>
> >>>     graph ?g1 {
> >>>>       ?thing a ?CLASS .
> >>>>
> >>> Unnecessary?
> > JMV: Yes, but removing it does not change the problem.
> >
> > AS:
> >
> >> ?g1 and ?CLASS aren't used so this is only some kind of existence test.
> It
> >> is a partial cross product on the first part.  With the DISTINCT in the
> >> SELECT there may a very large number of results yet only a few DISTINCT.
> >>
> >> JMV:
> >>
> >>
> >> Here is a simple query that is slow:
> > PREFIX text: <http://jena.apache.org/text#>
> > SELECT DISTINCT ?thing ?COUNT WHERE {
> >     graph ?g {
> >       ?thing text:query ( 'Jean-Marc' ) .
> >     }
> > } ORDER BY DESC(?COUNT)
> > LIMIT 10
> >
> >
> >
> >> AS:
> >>
> >>
> >> Remove the "graph ?g {".
> >>> (and then remove the DISTINCT)
> >>>
> >>> *JMV:*
> >>
> >> When I do this, the result is empty.
> >>> The way the text index is initialized must be wrong:
> >>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
> >>> forms/src/main/scala/deductions/runtime/jena/lucene/LuceneIndex.scala
> >>>
> > In fact, the getting the graph where where a literal triple belongs is
> not
> > relevant in this application.
> > So, reading again the doc.
> > https://jena.apache.org/documentation/query/text-
> query.html#graph-specific-indexing
> >
> > I understand that I must deactivate "graph-specific indexing" that is not
> > not needed, and slows the query.
> > The doc. does not say so, but I think rebuilding the index is
> appropriate.
> >
> >
> >
> >
> >
> >>>> JMV:
> >>>
> >> Run it with:
> >>>> time wget -O semantic-forms.cc_select-ui.txt
> >>>> http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A
> >>>> +%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+
> >>>> DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%
> >>>> 7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%
> >>>> 29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%
> 0D%0ALIMIT+10
> >>>>
> >>>> Or, if you want to use YasGUI , the endpoint is
> >>>> http://semantic-forms.cc:9112/sparql
> >>>>
> >>>> *Statistics on the database*
> >>>>
> >>>> 268 graphs and 588 864 triples.
> >>>>
> >>>> # Count graphs and triples
> >>>> SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
> >>>>       WHERE {
> >>>>     { GRAPH ?GR { } }
> >>>>     UNION
> >>>>     { GRAPH ?GR1 { ?s ?p ?o . } }
> >>>> }
> >>>>
> >>>> Result: 2 rows
> >>>> "grc" "trc"
> >>>> "268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
> >>>> http://www.w3.org/2001/XMLSchema#integer
> >>>>
> >>>> (I'm not sure this the right way to count, but it gives figures :) )
> >>>>
> >>>> You can reproduce the query with this UI :
> >>>>
> >>>> http://semantic-forms.cc:9112/select-ui?query=%23+Count+grap
> >>>> hs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%
> >>>> 28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+
> >>>> GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%
> >>>> 3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D
> >>>>
> >>>> This is using Jena 3.5.0. with TDB 1 .
> >>>>
> >> Here is a stack made with kill -3 when the app. was working hard;
> >>>> I put in bold a suspect line.
> >>>>
> >>>> "application-akka.actor.default-dispatcher-1351" #2683 prio=5
> os_prio=0
> >>>> tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
> >>>>      java.lang.Thread.State: RUNNABLE
> >>>> at sun.nio.ch.NativeThread.current(Native Method)
> >>>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
> >>>> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
> >>>> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
> >>>> at
> >>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.
> >>>> readInternal(NIOFSDirectory.java:179)
> >>>> at
> >>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIn
> >>>> dexInput.java:342)
> >>>> at
> >>>> org.apache.lucene.store.BufferedIndexInput.readByte(Buffered
> >>>> IndexInput.java:54)
> >>>> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
> >>>> at
> >>>> org.apache.lucene.store.BufferedIndexInput.readInt(BufferedI
> >>>> ndexInput.java:183)
> >>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
> >>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
> >>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
> >>>> at
> >>>> org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldR
> >>>> eader.java:91)
> >>>> at
> >>>> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<
> >>>> init>(BlockTreeTermsReader.java:234)
> >>>> at
> >>>> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fie
> >>>> ldsProducer(Lucene50PostingsFormat.java:445)
> >>>> at
> >>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
> >>>> ldsReader.<init>(PerFieldPostingsFormat.java:292)
> >>>> at
> >>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fie
> >>>> ldsProducer(PerFieldPostingsFormat.java:372)
> >>>> at
> >>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCor
> >>>> eReaders.java:112)
> >>>> at org.apache.lucene.index.SegmentReader.<init>(
> SegmentReader.java:74)
> >>>> at
> >>>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
> >>>> ndardDirectoryReader.java:62)
> >>>> at
> >>>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
> >>>> ndardDirectoryReader.java:54)
> >>>> at
> >>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(
> >>>> SegmentInfos.java:692)
> >>>> at
> >>>> org.apache.lucene.index.StandardDirectoryReader.open(Standar
> >>>> dDirectoryReader.java:77)
> >>>> at org.apache.lucene.index.DirectoryReader.open(
> DirectoryReader.java:63)
> >>>> at
> >>>> org.apache.jena.query.text.TextIndexLucene.query(TextIndexLu
> >>>> cene.java:370)
> >>>> at org.apache.jena.query.text.TextQueryPF.performQuery(TextQuer
> >>>> yPF.java:290)
> >>>> at
> >>>> org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQu
> >>>> eryPF.java:267)
> >>>> at
> >>>> org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189
> >>>> .call(Unknown
> >>>> Source)
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
> >>>> LocalManualCache$1.load(LocalCache.java:5065)
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$Loadi
> >>>> ngValueReference.loadFuture(LocalCache.java:3716)
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
> >>>> Segment.loadSync(LocalCache.java:2424)
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
> >>>> Segment.lockedGetOrLoad(LocalCache.java:2298)
> >>>> * - locked <0x00000000ef9d34f8> (a
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
> >>>> StrongAccessEntry)*
> >>>>
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
> >>>> Segment.get(LocalCache.java:2211)
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache.get(
> >>>> LocalCache.java:4154)
> >>>> at
> >>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
> >>>> LocalManualCache.get(LocalCache.java:5060)
> >>>> at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGu
> >>>> ava.java:58)
> >>>> at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
> >>>> at
> >>>> org.apache.jena.query.text.TextQueryPF.variableSubject(TextQ
> >>>> ueryPF.java:227)
> >>>> at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
> >>>> at
> >>>> org.apache.jena.sparql.pfunction.PropertyFunctionBase$Repeat
> >>>> ApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> >>>> makeNextStage(QueryIterRepeatApply.java:108)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> >>>> hasNextBinding(QueryIterRepeatApply.java:65)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterProcedure.ha
> >>>> sNextBinding(QueryIterProcedure.java:73)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterProcessBindi
> >>>> ng.hasNextBinding(QueryIterProcessBinding.java:66)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$Q
> >>>> ueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> >>>> hasNextBinding(QueryIterRepeatApply.java:74)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
> >>>> at
> >>>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(
> >>>> RepeatApplyIterator.java:45)
> >>>> at
> >>>> org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(
> >>>> SolverLib.java:195)
> >>>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
> >>>> .hasNextBinding(QueryIterPlainWrapper.java:53)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> >>>> makeNextStage(QueryIterRepeatApply.java:101)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> >>>> hasNextBinding(QueryIterRepeatApply.java:65)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasN
> >>>> extBinding(QueryIterConvert.java:58)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initi
> >>>> alizeIterator(QueryIterTopN.java:98)
> >>>> at
> >>>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
> >>>> .init(IteratorDelayedInitialization.java:40)
> >>>> at
> >>>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
> >>>> .hasNext(IteratorDelayedInitialization.java:50)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
> >>>> .hasNextBinding(QueryIterPlainWrapper.java:53)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
> >>>> hasNextBinding(QueryIteratorWrapper.java:39)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
> >>>> hasNextBinding(QueryIteratorWrapper.java:39)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
> >>>> hasNextBinding(QueryIteratorWrapper.java:39)
> >>>> at
> >>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> >>>> Next(QueryIteratorBase.java:114)
> >>>> at
> >>>> org.apache.jena.sparql.engine.ResultSetStream.hasNext(Result
> >>>> SetStream.java:74)
> >>>> at
> >>>> org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNex
> >>>> t(ResultSetCheckCondition.java:55)
> >>>> at
> >>>> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(
> >>>> Wrappers.scala:42)
> >>>> at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
> >>>> at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
> >>>> at
> >>>> scala.collection.TraversableOnce$class.toIterable(
> >>>> TraversableOnce.scala:296)
> >>>> at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
> >>>> at
> >>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
> >>>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
> >>>> at
> >>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
> >>>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
> >>>> at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
> >>>> at controllers.Application$.time(Application.scala:8)
> >>>> at
> >>>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
> >>>> lectQueryVariablesNT(SPARQLHelpers.scala:370)
> >>>> at
> >>>> controllers.Application$.sparqlSelectQueryVariablesNT(Applic
> >>>> ation.scala:8)
> >>>> at
> >>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
> >>>> apply(SPARQLHelpers.scala:361)
> >>>> at
> >>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
> >>>> apply(SPARQLHelpers.scala:361)
> >>>> at
> >>>> org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(
> >>>> JenaDatasetStore.scala:17)
> >>>> at scala.util.Try$.apply(Try.scala:192)
> >>>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
> >>>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
> >>>> at
> >>>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
> >>>> lectQueryVariables(SPARQLHelpers.scala:359)
> >>>> at controllers.Application$.sparqlSelectQueryVariables(Applicat
> >>>> ion.scala:8)
> >>>> at
> >>>> deductions.runtime.services.Lookup$class.searchStringOrClass
> >>>> (Lookup.scala:76)
> >>>> at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
> >>>> at controllers.Application$.lookup(Application.scala:8)
> >>>> at controllers.Services$$anonfun$lookupService$1.apply(Services
> >>>> .scala:199)
> >>>> at controllers.Services$$anonfun$lookupService$1.apply(Services
> >>>> .scala:193)
> >>>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.
> scala:371)
> >>>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.
> scala:370)
> >>>> at play.api.mvc.Action$.invokeBlock(Action.scala:498)
> >>>> at play.api.mvc.Action$.invokeBlock(Action.scala:495)
> >>>> at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
> >>>> at
> >>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
> >>>> un$apply$6.apply(Action.scala:112)
> >>>> at
> >>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
> >>>> un$apply$6.apply(Action.scala:112)
> >>>> at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
> >>>> at
> >>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
> >>>> Action.scala:111)
> >>>> at
> >>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
> >>>> Action.scala:110)
> >>>> at scala.Option.map(Option.scala:146)
> >>>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
> >>>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
> >>>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
> >>>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
> >>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> >>>> at
> >>>> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(Ba
> >>>> tchingExecutor.scala:55)
> >>>> at
> >>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
> >>>> 1.apply$mcV$sp(
> >>>> BatchingExecutor.scala:91)
> >>>> at
> >>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
> >>>> 1.apply(BatchingExecutor.scala:91)
> >>>> at
> >>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
> >>>> 1.apply(BatchingExecutor.scala:91)
> >>>> at scala.concurrent.BlockContext$.withBlockContext(BlockContext
> >>>> .scala:72)
> >>>> at
> >>>> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingEx
> >>>> ecutor.scala:90)
> >>>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> >>>> at
> >>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.
> >>>> exec(AbstractDispatcher.scala:415)
> >>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
> >>>> at
> >>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
> >>>> ForkJoinPool.java:1339)
> >>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPoo
> >>>> l.java:1979)
> >>>> at
> >>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinW
> >>>> orkerThread.java:107)
> >>>>
> >>>>
> >>>>
> >
>
>


-- 
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui

Re: Slow Lucene query

Posted by Lorenz Buehmann <bu...@informatik.uni-leipzig.de>.
>> Here is a simple query that is slow:
> PREFIX text: <http://jena.apache.org/text#>
> SELECT DISTINCT ?thing ?COUNT WHERE {
>     graph ?g {
>       ?thing text:query ( 'Jean-Marc' ) .
>     }
> } ORDER BY DESC(?COUNT)
> LIMIT 10
>
>
>
?COUNT is not bound here


On 05.12.2017 09:20, Jean-Marc Vanel wrote:
> 2017-12-04 10:21 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>> Hi - your messages loose the indication of who wrote what.
>>
> Sorry, I edited all , putting "AS:" and "JMV:" prefixes,
> and removing the less important stuff.
>
>
>> AS:
>>
>
>> Maybe it's this part of the query that causes some or all of the costs
>>
>>>     graph ?g1 {
>>>>       ?thing a ?CLASS .
>>>>
>>> Unnecessary?
> JMV: Yes, but removing it does not change the problem.
>
> AS:
>
>> ?g1 and ?CLASS aren't used so this is only some kind of existence test. It
>> is a partial cross product on the first part.  With the DISTINCT in the
>> SELECT there may a very large number of results yet only a few DISTINCT.
>>
>> JMV:
>>
>>
>> Here is a simple query that is slow:
> PREFIX text: <http://jena.apache.org/text#>
> SELECT DISTINCT ?thing ?COUNT WHERE {
>     graph ?g {
>       ?thing text:query ( 'Jean-Marc' ) .
>     }
> } ORDER BY DESC(?COUNT)
> LIMIT 10
>
>
>
>> AS:
>>
>>
>> Remove the "graph ?g {".
>>> (and then remove the DISTINCT)
>>>
>>> *JMV:*
>>
>> When I do this, the result is empty.
>>> The way the text index is initialized must be wrong:
>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>>> forms/src/main/scala/deductions/runtime/jena/lucene/LuceneIndex.scala
>>>
> In fact, the getting the graph where where a literal triple belongs is not
> relevant in this application.
> So, reading again the doc.
> https://jena.apache.org/documentation/query/text-query.html#graph-specific-indexing
>
> I understand that I must deactivate "graph-specific indexing" that is not
> not needed, and slows the query.
> The doc. does not say so, but I think rebuilding the index is appropriate.
>
>
>
>
>
>>>> JMV:
>>>
>> Run it with:
>>>> time wget -O semantic-forms.cc_select-ui.txt
>>>> http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A
>>>> +%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+
>>>> DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%
>>>> 7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%
>>>> 29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%0D%0ALIMIT+10
>>>>
>>>> Or, if you want to use YasGUI , the endpoint is
>>>> http://semantic-forms.cc:9112/sparql
>>>>
>>>> *Statistics on the database*
>>>>
>>>> 268 graphs and 588 864 triples.
>>>>
>>>> # Count graphs and triples
>>>> SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
>>>>       WHERE {
>>>>     { GRAPH ?GR { } }
>>>>     UNION
>>>>     { GRAPH ?GR1 { ?s ?p ?o . } }
>>>> }
>>>>
>>>> Result: 2 rows
>>>> "grc" "trc"
>>>> "268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
>>>> http://www.w3.org/2001/XMLSchema#integer
>>>>
>>>> (I'm not sure this the right way to count, but it gives figures :) )
>>>>
>>>> You can reproduce the query with this UI :
>>>>
>>>> http://semantic-forms.cc:9112/select-ui?query=%23+Count+grap
>>>> hs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%
>>>> 28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+
>>>> GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%
>>>> 3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D
>>>>
>>>> This is using Jena 3.5.0. with TDB 1 .
>>>>
>> Here is a stack made with kill -3 when the app. was working hard;
>>>> I put in bold a suspect line.
>>>>
>>>> "application-akka.actor.default-dispatcher-1351" #2683 prio=5 os_prio=0
>>>> tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
>>>>      java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.NativeThread.current(Native Method)
>>>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
>>>> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
>>>> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
>>>> at
>>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.
>>>> readInternal(NIOFSDirectory.java:179)
>>>> at
>>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIn
>>>> dexInput.java:342)
>>>> at
>>>> org.apache.lucene.store.BufferedIndexInput.readByte(Buffered
>>>> IndexInput.java:54)
>>>> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
>>>> at
>>>> org.apache.lucene.store.BufferedIndexInput.readInt(BufferedI
>>>> ndexInput.java:183)
>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
>>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
>>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
>>>> at
>>>> org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldR
>>>> eader.java:91)
>>>> at
>>>> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<
>>>> init>(BlockTreeTermsReader.java:234)
>>>> at
>>>> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fie
>>>> ldsProducer(Lucene50PostingsFormat.java:445)
>>>> at
>>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
>>>> ldsReader.<init>(PerFieldPostingsFormat.java:292)
>>>> at
>>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fie
>>>> ldsProducer(PerFieldPostingsFormat.java:372)
>>>> at
>>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCor
>>>> eReaders.java:112)
>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
>>>> at
>>>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
>>>> ndardDirectoryReader.java:62)
>>>> at
>>>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
>>>> ndardDirectoryReader.java:54)
>>>> at
>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(
>>>> SegmentInfos.java:692)
>>>> at
>>>> org.apache.lucene.index.StandardDirectoryReader.open(Standar
>>>> dDirectoryReader.java:77)
>>>> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
>>>> at
>>>> org.apache.jena.query.text.TextIndexLucene.query(TextIndexLu
>>>> cene.java:370)
>>>> at org.apache.jena.query.text.TextQueryPF.performQuery(TextQuer
>>>> yPF.java:290)
>>>> at
>>>> org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQu
>>>> eryPF.java:267)
>>>> at
>>>> org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189
>>>> .call(Unknown
>>>> Source)
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>>> LocalManualCache$1.load(LocalCache.java:5065)
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$Loadi
>>>> ngValueReference.loadFuture(LocalCache.java:3716)
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>>> Segment.loadSync(LocalCache.java:2424)
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>>> Segment.lockedGetOrLoad(LocalCache.java:2298)
>>>> * - locked <0x00000000ef9d34f8> (a
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>>> StrongAccessEntry)*
>>>>
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>>> Segment.get(LocalCache.java:2211)
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache.get(
>>>> LocalCache.java:4154)
>>>> at
>>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>>> LocalManualCache.get(LocalCache.java:5060)
>>>> at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGu
>>>> ava.java:58)
>>>> at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
>>>> at
>>>> org.apache.jena.query.text.TextQueryPF.variableSubject(TextQ
>>>> ueryPF.java:227)
>>>> at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
>>>> at
>>>> org.apache.jena.sparql.pfunction.PropertyFunctionBase$Repeat
>>>> ApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>>> makeNextStage(QueryIterRepeatApply.java:108)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>>> hasNextBinding(QueryIterRepeatApply.java:65)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterProcedure.ha
>>>> sNextBinding(QueryIterProcedure.java:73)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterProcessBindi
>>>> ng.hasNextBinding(QueryIterProcessBinding.java:66)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$Q
>>>> ueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>>> hasNextBinding(QueryIterRepeatApply.java:74)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
>>>> at
>>>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(
>>>> RepeatApplyIterator.java:45)
>>>> at
>>>> org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(
>>>> SolverLib.java:195)
>>>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
>>>> .hasNextBinding(QueryIterPlainWrapper.java:53)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>>> makeNextStage(QueryIterRepeatApply.java:101)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>>> hasNextBinding(QueryIterRepeatApply.java:65)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasN
>>>> extBinding(QueryIterConvert.java:58)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initi
>>>> alizeIterator(QueryIterTopN.java:98)
>>>> at
>>>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
>>>> .init(IteratorDelayedInitialization.java:40)
>>>> at
>>>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
>>>> .hasNext(IteratorDelayedInitialization.java:50)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
>>>> .hasNextBinding(QueryIterPlainWrapper.java:53)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>>>> hasNextBinding(QueryIteratorWrapper.java:39)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>>>> hasNextBinding(QueryIteratorWrapper.java:39)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>>>> hasNextBinding(QueryIteratorWrapper.java:39)
>>>> at
>>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>>> Next(QueryIteratorBase.java:114)
>>>> at
>>>> org.apache.jena.sparql.engine.ResultSetStream.hasNext(Result
>>>> SetStream.java:74)
>>>> at
>>>> org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNex
>>>> t(ResultSetCheckCondition.java:55)
>>>> at
>>>> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(
>>>> Wrappers.scala:42)
>>>> at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
>>>> at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
>>>> at
>>>> scala.collection.TraversableOnce$class.toIterable(
>>>> TraversableOnce.scala:296)
>>>> at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
>>>> at
>>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
>>>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
>>>> at
>>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
>>>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
>>>> at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
>>>> at controllers.Application$.time(Application.scala:8)
>>>> at
>>>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
>>>> lectQueryVariablesNT(SPARQLHelpers.scala:370)
>>>> at
>>>> controllers.Application$.sparqlSelectQueryVariablesNT(Applic
>>>> ation.scala:8)
>>>> at
>>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
>>>> apply(SPARQLHelpers.scala:361)
>>>> at
>>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
>>>> apply(SPARQLHelpers.scala:361)
>>>> at
>>>> org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(
>>>> JenaDatasetStore.scala:17)
>>>> at scala.util.Try$.apply(Try.scala:192)
>>>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
>>>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
>>>> at
>>>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
>>>> lectQueryVariables(SPARQLHelpers.scala:359)
>>>> at controllers.Application$.sparqlSelectQueryVariables(Applicat
>>>> ion.scala:8)
>>>> at
>>>> deductions.runtime.services.Lookup$class.searchStringOrClass
>>>> (Lookup.scala:76)
>>>> at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
>>>> at controllers.Application$.lookup(Application.scala:8)
>>>> at controllers.Services$$anonfun$lookupService$1.apply(Services
>>>> .scala:199)
>>>> at controllers.Services$$anonfun$lookupService$1.apply(Services
>>>> .scala:193)
>>>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>>>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
>>>> at play.api.mvc.Action$.invokeBlock(Action.scala:498)
>>>> at play.api.mvc.Action$.invokeBlock(Action.scala:495)
>>>> at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
>>>> at
>>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
>>>> un$apply$6.apply(Action.scala:112)
>>>> at
>>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
>>>> un$apply$6.apply(Action.scala:112)
>>>> at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
>>>> at
>>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
>>>> Action.scala:111)
>>>> at
>>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
>>>> Action.scala:110)
>>>> at scala.Option.map(Option.scala:146)
>>>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
>>>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
>>>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
>>>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
>>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
>>>> at
>>>> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(Ba
>>>> tchingExecutor.scala:55)
>>>> at
>>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>>>> 1.apply$mcV$sp(
>>>> BatchingExecutor.scala:91)
>>>> at
>>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>>>> 1.apply(BatchingExecutor.scala:91)
>>>> at
>>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>>>> 1.apply(BatchingExecutor.scala:91)
>>>> at scala.concurrent.BlockContext$.withBlockContext(BlockContext
>>>> .scala:72)
>>>> at
>>>> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingEx
>>>> ecutor.scala:90)
>>>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
>>>> at
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.
>>>> exec(AbstractDispatcher.scala:415)
>>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
>>>> ForkJoinPool.java:1339)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPoo
>>>> l.java:1979)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinW
>>>> orkerThread.java:107)
>>>>
>>>>
>>>>
>

Re: Slow Lucene query

Posted by Jean-Marc Vanel <je...@gmail.com>.
2017-12-04 10:21 GMT+01:00 Andy Seaborne <an...@apache.org>:

> Hi - your messages loose the indication of who wrote what.
>

Sorry, I edited all , putting "AS:" and "JMV:" prefixes,
and removing the less important stuff.


> AS:
>


> Maybe it's this part of the query that causes some or all of the costs
>
> >     graph ?g1 {
> >>       ?thing a ?CLASS .
> >>
> >
> > Unnecessary?
>

JMV: Yes, but removing it does not change the problem.

AS:
>


> ?g1 and ?CLASS aren't used so this is only some kind of existence test. It
> is a partial cross product on the first part.  With the DISTINCT in the
> SELECT there may a very large number of results yet only a few DISTINCT.
>
> JMV:
>
>

> Here is a simple query that is slow:
>>
>
PREFIX text: <http://jena.apache.org/text#>
SELECT DISTINCT ?thing ?COUNT WHERE {
    graph ?g {
      ?thing text:query ( 'Jean-Marc' ) .
    }
} ORDER BY DESC(?COUNT)
LIMIT 10



> AS:
>
>

> Remove the "graph ?g {".
>>
>> (and then remove the DISTINCT)
>>
>> *JMV:*
>
>

> When I do this, the result is empty.
>>
>> The way the text index is initialized must be wrong:
>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>> forms/src/main/scala/deductions/runtime/jena/lucene/LuceneIndex.scala
>>
>
In fact, the getting the graph where where a literal triple belongs is not
relevant in this application.
So, reading again the doc.
https://jena.apache.org/documentation/query/text-query.html#graph-specific-indexing

I understand that I must deactivate "graph-specific indexing" that is not
not needed, and slows the query.
The doc. does not say so, but I think rebuilding the index is appropriate.





>>> JMV:
>>
>>

> Run it with:
>>> time wget -O semantic-forms.cc_select-ui.txt
>>> http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A
>>> +%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+
>>> DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%
>>> 7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%
>>> 29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%0D%0ALIMIT+10
>>>
>>> Or, if you want to use YasGUI , the endpoint is
>>> http://semantic-forms.cc:9112/sparql
>>>
>>> *Statistics on the database*
>>>
>>> 268 graphs and 588 864 triples.
>>>
>>> # Count graphs and triples
>>> SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
>>>       WHERE {
>>>     { GRAPH ?GR { } }
>>>     UNION
>>>     { GRAPH ?GR1 { ?s ?p ?o . } }
>>> }
>>>
>>> Result: 2 rows
>>> "grc" "trc"
>>> "268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
>>> http://www.w3.org/2001/XMLSchema#integer
>>>
>>> (I'm not sure this the right way to count, but it gives figures :) )
>>>
>>> You can reproduce the query with this UI :
>>>
>>> http://semantic-forms.cc:9112/select-ui?query=%23+Count+grap
>>> hs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%
>>> 28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+
>>> GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%
>>> 3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D
>>>
>>> This is using Jena 3.5.0. with TDB 1 .
>>>
>>

> Here is a stack made with kill -3 when the app. was working hard;
>>> I put in bold a suspect line.
>>>
>>> "application-akka.actor.default-dispatcher-1351" #2683 prio=5 os_prio=0
>>> tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
>>>      java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.NativeThread.current(Native Method)
>>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
>>> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
>>> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
>>> at
>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.
>>> readInternal(NIOFSDirectory.java:179)
>>> at
>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIn
>>> dexInput.java:342)
>>> at
>>> org.apache.lucene.store.BufferedIndexInput.readByte(Buffered
>>> IndexInput.java:54)
>>> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
>>> at
>>> org.apache.lucene.store.BufferedIndexInput.readInt(BufferedI
>>> ndexInput.java:183)
>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
>>> at
>>> org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldR
>>> eader.java:91)
>>> at
>>> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<
>>> init>(BlockTreeTermsReader.java:234)
>>> at
>>> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fie
>>> ldsProducer(Lucene50PostingsFormat.java:445)
>>> at
>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
>>> ldsReader.<init>(PerFieldPostingsFormat.java:292)
>>> at
>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fie
>>> ldsProducer(PerFieldPostingsFormat.java:372)
>>> at
>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCor
>>> eReaders.java:112)
>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
>>> at
>>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
>>> ndardDirectoryReader.java:62)
>>> at
>>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
>>> ndardDirectoryReader.java:54)
>>> at
>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(
>>> SegmentInfos.java:692)
>>> at
>>> org.apache.lucene.index.StandardDirectoryReader.open(Standar
>>> dDirectoryReader.java:77)
>>> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
>>> at
>>> org.apache.jena.query.text.TextIndexLucene.query(TextIndexLu
>>> cene.java:370)
>>> at org.apache.jena.query.text.TextQueryPF.performQuery(TextQuer
>>> yPF.java:290)
>>> at
>>> org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQu
>>> eryPF.java:267)
>>> at
>>> org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189
>>> .call(Unknown
>>> Source)
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>> LocalManualCache$1.load(LocalCache.java:5065)
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$Loadi
>>> ngValueReference.loadFuture(LocalCache.java:3716)
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>> Segment.loadSync(LocalCache.java:2424)
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>> Segment.lockedGetOrLoad(LocalCache.java:2298)
>>> * - locked <0x00000000ef9d34f8> (a
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>> StrongAccessEntry)*
>>>
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>> Segment.get(LocalCache.java:2211)
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache.get(
>>> LocalCache.java:4154)
>>> at
>>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>>> LocalManualCache.get(LocalCache.java:5060)
>>> at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGu
>>> ava.java:58)
>>> at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
>>> at
>>> org.apache.jena.query.text.TextQueryPF.variableSubject(TextQ
>>> ueryPF.java:227)
>>> at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
>>> at
>>> org.apache.jena.sparql.pfunction.PropertyFunctionBase$Repeat
>>> ApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>> makeNextStage(QueryIterRepeatApply.java:108)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>> hasNextBinding(QueryIterRepeatApply.java:65)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterProcedure.ha
>>> sNextBinding(QueryIterProcedure.java:73)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterProcessBindi
>>> ng.hasNextBinding(QueryIterProcessBinding.java:66)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$Q
>>> ueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>> hasNextBinding(QueryIterRepeatApply.java:74)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
>>> at
>>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(
>>> RepeatApplyIterator.java:45)
>>> at
>>> org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(
>>> SolverLib.java:195)
>>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
>>> .hasNextBinding(QueryIterPlainWrapper.java:53)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>> makeNextStage(QueryIterRepeatApply.java:101)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>>> hasNextBinding(QueryIterRepeatApply.java:65)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasN
>>> extBinding(QueryIterConvert.java:58)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initi
>>> alizeIterator(QueryIterTopN.java:98)
>>> at
>>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
>>> .init(IteratorDelayedInitialization.java:40)
>>> at
>>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
>>> .hasNext(IteratorDelayedInitialization.java:50)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
>>> .hasNextBinding(QueryIterPlainWrapper.java:53)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>>> hasNextBinding(QueryIteratorWrapper.java:39)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>>> hasNextBinding(QueryIteratorWrapper.java:39)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>>> hasNextBinding(QueryIteratorWrapper.java:39)
>>> at
>>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>>> Next(QueryIteratorBase.java:114)
>>> at
>>> org.apache.jena.sparql.engine.ResultSetStream.hasNext(Result
>>> SetStream.java:74)
>>> at
>>> org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNex
>>> t(ResultSetCheckCondition.java:55)
>>> at
>>> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(
>>> Wrappers.scala:42)
>>> at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
>>> at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
>>> at
>>> scala.collection.TraversableOnce$class.toIterable(
>>> TraversableOnce.scala:296)
>>> at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
>>> at
>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
>>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
>>> at
>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
>>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
>>> at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
>>> at controllers.Application$.time(Application.scala:8)
>>> at
>>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
>>> lectQueryVariablesNT(SPARQLHelpers.scala:370)
>>> at
>>> controllers.Application$.sparqlSelectQueryVariablesNT(Applic
>>> ation.scala:8)
>>> at
>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
>>> apply(SPARQLHelpers.scala:361)
>>> at
>>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
>>> apply(SPARQLHelpers.scala:361)
>>> at
>>> org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(
>>> JenaDatasetStore.scala:17)
>>> at scala.util.Try$.apply(Try.scala:192)
>>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
>>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
>>> at
>>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
>>> lectQueryVariables(SPARQLHelpers.scala:359)
>>> at controllers.Application$.sparqlSelectQueryVariables(Applicat
>>> ion.scala:8)
>>> at
>>> deductions.runtime.services.Lookup$class.searchStringOrClass
>>> (Lookup.scala:76)
>>> at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
>>> at controllers.Application$.lookup(Application.scala:8)
>>> at controllers.Services$$anonfun$lookupService$1.apply(Services
>>> .scala:199)
>>> at controllers.Services$$anonfun$lookupService$1.apply(Services
>>> .scala:193)
>>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
>>> at play.api.mvc.Action$.invokeBlock(Action.scala:498)
>>> at play.api.mvc.Action$.invokeBlock(Action.scala:495)
>>> at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
>>> at
>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
>>> un$apply$6.apply(Action.scala:112)
>>> at
>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
>>> un$apply$6.apply(Action.scala:112)
>>> at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
>>> at
>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
>>> Action.scala:111)
>>> at
>>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
>>> Action.scala:110)
>>> at scala.Option.map(Option.scala:146)
>>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
>>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
>>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
>>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
>>> at
>>> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(Ba
>>> tchingExecutor.scala:55)
>>> at
>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>>> 1.apply$mcV$sp(
>>> BatchingExecutor.scala:91)
>>> at
>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>>> 1.apply(BatchingExecutor.scala:91)
>>> at
>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>>> 1.apply(BatchingExecutor.scala:91)
>>> at scala.concurrent.BlockContext$.withBlockContext(BlockContext
>>> .scala:72)
>>> at
>>> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingEx
>>> ecutor.scala:90)
>>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
>>> at
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.
>>> exec(AbstractDispatcher.scala:415)
>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>> at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
>>> ForkJoinPool.java:1339)
>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPoo
>>> l.java:1979)
>>> at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinW
>>> orkerThread.java:107)
>>>
>>>
>>>
>>


-- 
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui

Re: Slow Lucene query

Posted by Andy Seaborne <an...@apache.org>.
Hi - your messages loose the indication of who wrote what.

Maybe it's this part of the query that causes some or all of the costs

 >     graph ?g1 {
 >>       ?thing a ?CLASS .
 >>
 >
 > Unnecessary?
 >

?g1 and ?CLASS aren't used so this is only some kind of existence test. 
It is a partial cross product on the first part.  With the DISTINCT in 
the SELECT there may a very large number of results yet only a few DISTINCT.

     Andy

On 04/12/17 08:55, Jean-Marc Vanel wrote:
> Le 1 déc. 2017 8:08 PM, "Andy Seaborne" <an...@apache.org> a écrit :
> 
> 
> 
> On 01/12/17 11:49, Jean-Marc Vanel wrote:
> 
>> Hi
>>
>> The time for a SPARQL + Lucene query is more than 1 mn.
>> It used to be around one second, and then the database grew.
>> Here is a typical query run for a lookup service similar to dbpedia lookup:
>>
>> PREFIX text: <http://jena.apache.org/text#>
>> PREFIX form: <
>> http://raw.githubusercontent.com/jmvanel/semantic_forms/mast
>> er/vocabulary/forms.owl.ttl#
>>
>>>
>>> SELECT DISTINCT ?thing ?COUNT WHERE {
>>     graph ?g {
>>       ?thing text:query ( 'Jean-Marc' ) .
>>     }
>>
> 
> This is going to loop on each graph and make a text:query for each one.
> 
> Is that what you intended?
> 
> Remove the "graph ?g {".
> 
> (and then remove the DISTINCT)
> 
> When I do this, the result is empty.
> 
> The way the text index is initialized must be wrong:
> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms/src/main/scala/deductions/runtime/jena/lucene/LuceneIndex.scala
> On fact, the graph where where a literal triple belongs is not relevant un
> this application.
> 
> 
>     graph ?g1 {
>>       ?thing a ?CLASS .
>>
> 
> Unnecessary?
> 
>     }
>>     OPTIONAL {
>>      graph ?grCount {
>>       ?thing form:linksCount ?COUNT.
>>     } }
>> } ORDER BY DESC(?COUNT)
>> LIMIT 10
>>
>> Here is a simpler query that is also slow:
>>
>> PREFIX text: <http://jena.apache.org/text#>
>> SELECT DISTINCT ?thing ?COUNT WHERE {
>>     graph ?g {
>>       ?thing text:query ( 'Jean-Marc' ) .
>>     }
>> } ORDER BY DESC(?COUNT)
>> LIMIT 10
>>
>> Run it with:
>> time wget -O semantic-forms.cc_select-ui.txt
>> http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A
>> +%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+
>> DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%
>> 7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%
>> 29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%0D%0ALIMIT+10
>>
>> Or, if you want to use YasGUI , the endpoint is
>> http://semantic-forms.cc:9112/sparql
>>
>> *Statistics on the database*
>>
>> 268 graphs and 588 864 triples.
>>
>> # Count graphs and triples
>> SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
>>       WHERE {
>>     { GRAPH ?GR { } }
>>     UNION
>>     { GRAPH ?GR1 { ?s ?p ?o . } }
>> }
>>
>> Result: 2 rows
>> "grc" "trc"
>> "268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
>> http://www.w3.org/2001/XMLSchema#integer
>>
>> (I'm not sure this the right way to count, but it gives figures :) )
>>
>> You can reproduce the query with this UI :
>>
>> http://semantic-forms.cc:9112/select-ui?query=%23+Count+grap
>> hs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%
>> 28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+
>> GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%
>> 3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D
>>
>> This is using Jena 3.5.0. with TDB 1 .
>> Here is a stack made with kill -3 when the app. was working hard;
>> I put in bold a suspect line.
>>
>> "application-akka.actor.default-dispatcher-1351" #2683 prio=5 os_prio=0
>> tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
>>      java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.NativeThread.current(Native Method)
>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
>> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
>> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
>> at
>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.
>> readInternal(NIOFSDirectory.java:179)
>> at
>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIn
>> dexInput.java:342)
>> at
>> org.apache.lucene.store.BufferedIndexInput.readByte(Buffered
>> IndexInput.java:54)
>> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
>> at
>> org.apache.lucene.store.BufferedIndexInput.readInt(BufferedI
>> ndexInput.java:183)
>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
>> at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
>> at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
>> at
>> org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldReader.java:91)
>> at
>> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<
>> init>(BlockTreeTermsReader.java:234)
>> at
>> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fie
>> ldsProducer(Lucene50PostingsFormat.java:445)
>> at
>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
>> ldsReader.<init>(PerFieldPostingsFormat.java:292)
>> at
>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fie
>> ldsProducer(PerFieldPostingsFormat.java:372)
>> at
>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCor
>> eReaders.java:112)
>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
>> at
>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
>> ndardDirectoryReader.java:62)
>> at
>> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
>> ndardDirectoryReader.java:54)
>> at
>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(
>> SegmentInfos.java:692)
>> at
>> org.apache.lucene.index.StandardDirectoryReader.open(Standar
>> dDirectoryReader.java:77)
>> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
>> at
>> org.apache.jena.query.text.TextIndexLucene.query(TextIndexLucene.java:370)
>> at org.apache.jena.query.text.TextQueryPF.performQuery(TextQuer
>> yPF.java:290)
>> at
>> org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQu
>> eryPF.java:267)
>> at
>> org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189.call(Unknown
>> Source)
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>> LocalManualCache$1.load(LocalCache.java:5065)
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache$Loadi
>> ngValueReference.loadFuture(LocalCache.java:3716)
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>> Segment.loadSync(LocalCache.java:2424)
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>> Segment.lockedGetOrLoad(LocalCache.java:2298)
>> * - locked <0x00000000ef9d34f8> (a
>> org.apache.jena.ext.com.google.common.cache.LocalCache$StrongAccessEntry)*
>>
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>> Segment.get(LocalCache.java:2211)
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache.get(
>> LocalCache.java:4154)
>> at
>> org.apache.jena.ext.com.google.common.cache.LocalCache$
>> LocalManualCache.get(LocalCache.java:5060)
>> at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGu
>> ava.java:58)
>> at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
>> at
>> org.apache.jena.query.text.TextQueryPF.variableSubject(TextQ
>> ueryPF.java:227)
>> at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
>> at
>> org.apache.jena.sparql.pfunction.PropertyFunctionBase$Repeat
>> ApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>> makeNextStage(QueryIterRepeatApply.java:108)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>> hasNextBinding(QueryIterRepeatApply.java:65)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterProcedure.ha
>> sNextBinding(QueryIterProcedure.java:73)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterProcessBindi
>> ng.hasNextBinding(QueryIterProcessBinding.java:66)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$Q
>> ueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>> hasNextBinding(QueryIterRepeatApply.java:74)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
>> at
>> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(
>> RepeatApplyIterator.java:45)
>> at
>> org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(
>> SolverLib.java:195)
>> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
>> .hasNextBinding(QueryIterPlainWrapper.java:53)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>> makeNextStage(QueryIterRepeatApply.java:101)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
>> hasNextBinding(QueryIterRepeatApply.java:65)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasN
>> extBinding(QueryIterConvert.java:58)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initi
>> alizeIterator(QueryIterTopN.java:98)
>> at
>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
>> .init(IteratorDelayedInitialization.java:40)
>> at
>> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
>> .hasNext(IteratorDelayedInitialization.java:50)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
>> .hasNextBinding(QueryIterPlainWrapper.java:53)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>> hasNextBinding(QueryIteratorWrapper.java:39)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>> hasNextBinding(QueryIteratorWrapper.java:39)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
>> hasNextBinding(QueryIteratorWrapper.java:39)
>> at
>> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
>> Next(QueryIteratorBase.java:114)
>> at
>> org.apache.jena.sparql.engine.ResultSetStream.hasNext(Result
>> SetStream.java:74)
>> at
>> org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNex
>> t(ResultSetCheckCondition.java:55)
>> at
>> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(
>> Wrappers.scala:42)
>> at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
>> at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
>> at
>> scala.collection.TraversableOnce$class.toIterable(
>> TraversableOnce.scala:296)
>> at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
>> at
>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
>> at
>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
>> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
>> at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
>> at controllers.Application$.time(Application.scala:8)
>> at
>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
>> lectQueryVariablesNT(SPARQLHelpers.scala:370)
>> at
>> controllers.Application$.sparqlSelectQueryVariablesNT(Application.scala:8)
>> at
>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
>> apply(SPARQLHelpers.scala:361)
>> at
>> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
>> apply(SPARQLHelpers.scala:361)
>> at
>> org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(
>> JenaDatasetStore.scala:17)
>> at scala.util.Try$.apply(Try.scala:192)
>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
>> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
>> at
>> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
>> lectQueryVariables(SPARQLHelpers.scala:359)
>> at controllers.Application$.sparqlSelectQueryVariables(Applicat
>> ion.scala:8)
>> at
>> deductions.runtime.services.Lookup$class.searchStringOrClass
>> (Lookup.scala:76)
>> at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
>> at controllers.Application$.lookup(Application.scala:8)
>> at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:199)
>> at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:193)
>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
>> at play.api.mvc.Action$.invokeBlock(Action.scala:498)
>> at play.api.mvc.Action$.invokeBlock(Action.scala:495)
>> at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
>> at
>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
>> un$apply$6.apply(Action.scala:112)
>> at
>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
>> un$apply$6.apply(Action.scala:112)
>> at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
>> at
>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
>> Action.scala:111)
>> at
>> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
>> Action.scala:110)
>> at scala.Option.map(Option.scala:146)
>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
>> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
>> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
>> at
>> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(Ba
>> tchingExecutor.scala:55)
>> at
>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(
>> BatchingExecutor.scala:91)
>> at
>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>> 1.apply(BatchingExecutor.scala:91)
>> at
>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
>> 1.apply(BatchingExecutor.scala:91)
>> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>> at
>> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingEx
>> ecutor.scala:90)
>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
>> at
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.
>> exec(AbstractDispatcher.scala:415)
>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
>> ForkJoinPool.java:1339)
>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPoo
>> l.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinW
>> orkerThread.java:107)
>>
>>
>

Re: Slow Lucene query

Posted by Jean-Marc Vanel <je...@gmail.com>.
Le 1 déc. 2017 8:08 PM, "Andy Seaborne" <an...@apache.org> a écrit :



On 01/12/17 11:49, Jean-Marc Vanel wrote:

> Hi
>
> The time for a SPARQL + Lucene query is more than 1 mn.
> It used to be around one second, and then the database grew.
> Here is a typical query run for a lookup service similar to dbpedia lookup:
>
> PREFIX text: <http://jena.apache.org/text#>
> PREFIX form: <
> http://raw.githubusercontent.com/jmvanel/semantic_forms/mast
> er/vocabulary/forms.owl.ttl#
>
>>
>> SELECT DISTINCT ?thing ?COUNT WHERE {
>    graph ?g {
>      ?thing text:query ( 'Jean-Marc' ) .
>    }
>

This is going to loop on each graph and make a text:query for each one.

Is that what you intended?

Remove the "graph ?g {".

(and then remove the DISTINCT)

When I do this, the result is empty.

The way the text index is initialized must be wrong:
https://github.com/jmvanel/semantic_forms/blob/master/scala/forms/src/main/scala/deductions/runtime/jena/lucene/LuceneIndex.scala
On fact, the graph where where a literal triple belongs is not relevant un
this application.


   graph ?g1 {
>      ?thing a ?CLASS .
>

Unnecessary?

   }
>    OPTIONAL {
>     graph ?grCount {
>      ?thing form:linksCount ?COUNT.
>    } }
> } ORDER BY DESC(?COUNT)
> LIMIT 10
>
> Here is a simpler query that is also slow:
>
> PREFIX text: <http://jena.apache.org/text#>
> SELECT DISTINCT ?thing ?COUNT WHERE {
>    graph ?g {
>      ?thing text:query ( 'Jean-Marc' ) .
>    }
> } ORDER BY DESC(?COUNT)
> LIMIT 10
>
> Run it with:
> time wget -O semantic-forms.cc_select-ui.txt
> http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A
> +%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+
> DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%
> 7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%
> 29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%0D%0ALIMIT+10
>
> Or, if you want to use YasGUI , the endpoint is
> http://semantic-forms.cc:9112/sparql
>
> *Statistics on the database*
>
> 268 graphs and 588 864 triples.
>
> # Count graphs and triples
> SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
>      WHERE {
>    { GRAPH ?GR { } }
>    UNION
>    { GRAPH ?GR1 { ?s ?p ?o . } }
> }
>
> Result: 2 rows
> "grc" "trc"
> "268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
> http://www.w3.org/2001/XMLSchema#integer
>
> (I'm not sure this the right way to count, but it gives figures :) )
>
> You can reproduce the query with this UI :
>
> http://semantic-forms.cc:9112/select-ui?query=%23+Count+grap
> hs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%
> 28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+
> GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%
> 3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D
>
> This is using Jena 3.5.0. with TDB 1 .
> Here is a stack made with kill -3 when the app. was working hard;
> I put in bold a suspect line.
>
> "application-akka.actor.default-dispatcher-1351" #2683 prio=5 os_prio=0
> tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
>     java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
> at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.
> readInternal(NIOFSDirectory.java:179)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIn
> dexInput.java:342)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(Buffered
> IndexInput.java:54)
> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
> at
> org.apache.lucene.store.BufferedIndexInput.readInt(BufferedI
> ndexInput.java:183)
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
> at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
> at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
> at
> org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldReader.java:91)
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<
> init>(BlockTreeTermsReader.java:234)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fie
> ldsProducer(Lucene50PostingsFormat.java:445)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
> ldsReader.<init>(PerFieldPostingsFormat.java:292)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fie
> ldsProducer(PerFieldPostingsFormat.java:372)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCor
> eReaders.java:112)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
> ndardDirectoryReader.java:62)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(Sta
> ndardDirectoryReader.java:54)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(
> SegmentInfos.java:692)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(Standar
> dDirectoryReader.java:77)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
> at
> org.apache.jena.query.text.TextIndexLucene.query(TextIndexLucene.java:370)
> at org.apache.jena.query.text.TextQueryPF.performQuery(TextQuer
> yPF.java:290)
> at
> org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQu
> eryPF.java:267)
> at
> org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189.call(Unknown
> Source)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$
> LocalManualCache$1.load(LocalCache.java:5065)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$Loadi
> ngValueReference.loadFuture(LocalCache.java:3716)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$
> Segment.loadSync(LocalCache.java:2424)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$
> Segment.lockedGetOrLoad(LocalCache.java:2298)
> * - locked <0x00000000ef9d34f8> (a
> org.apache.jena.ext.com.google.common.cache.LocalCache$StrongAccessEntry)*
>
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$
> Segment.get(LocalCache.java:2211)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache.get(
> LocalCache.java:4154)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$
> LocalManualCache.get(LocalCache.java:5060)
> at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGu
> ava.java:58)
> at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
> at
> org.apache.jena.query.text.TextQueryPF.variableSubject(TextQ
> ueryPF.java:227)
> at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
> at
> org.apache.jena.sparql.pfunction.PropertyFunctionBase$Repeat
> ApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> makeNextStage(QueryIterRepeatApply.java:108)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> hasNextBinding(QueryIterRepeatApply.java:65)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterProcedure.ha
> sNextBinding(QueryIterProcedure.java:73)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterProcessBindi
> ng.hasNextBinding(QueryIterProcessBinding.java:66)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$Q
> ueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> hasNextBinding(QueryIterRepeatApply.java:74)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
> at
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(
> RepeatApplyIterator.java:45)
> at
> org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(
> SolverLib.java:195)
> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
> .hasNextBinding(QueryIterPlainWrapper.java:53)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> makeNextStage(QueryIterRepeatApply.java:101)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.
> hasNextBinding(QueryIterRepeatApply.java:65)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasN
> extBinding(QueryIterConvert.java:58)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initi
> alizeIterator(QueryIterTopN.java:98)
> at
> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
> .init(IteratorDelayedInitialization.java:40)
> at
> org.apache.jena.atlas.iterator.IteratorDelayedInitialization
> .hasNext(IteratorDelayedInitialization.java:50)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper
> .hasNextBinding(QueryIterPlainWrapper.java:53)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
> hasNextBinding(QueryIteratorWrapper.java:39)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
> hasNextBinding(QueryIteratorWrapper.java:39)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.
> hasNextBinding(QueryIteratorWrapper.java:39)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.has
> Next(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.ResultSetStream.hasNext(Result
> SetStream.java:74)
> at
> org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNex
> t(ResultSetCheckCondition.java:55)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(
> Wrappers.scala:42)
> at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
> at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
> at
> scala.collection.TraversableOnce$class.toIterable(
> TraversableOnce.scala:296)
> at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparq
> lSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
> at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
> at controllers.Application$.time(Application.scala:8)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
> lectQueryVariablesNT(SPARQLHelpers.scala:370)
> at
> controllers.Application$.sparqlSelectQueryVariablesNT(Application.scala:8)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
> apply(SPARQLHelpers.scala:361)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.
> apply(SPARQLHelpers.scala:361)
> at
> org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(
> JenaDatasetStore.scala:17)
> at scala.util.Try$.apply(Try.scala:192)
> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSe
> lectQueryVariables(SPARQLHelpers.scala:359)
> at controllers.Application$.sparqlSelectQueryVariables(Applicat
> ion.scala:8)
> at
> deductions.runtime.services.Lookup$class.searchStringOrClass
> (Lookup.scala:76)
> at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
> at controllers.Application$.lookup(Application.scala:8)
> at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:199)
> at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:193)
> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
> at play.api.mvc.Action$.invokeBlock(Action.scala:498)
> at play.api.mvc.Action$.invokeBlock(Action.scala:495)
> at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
> un$apply$6.apply(Action.scala:112)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonf
> un$apply$6.apply(Action.scala:112)
> at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
> Action.scala:111)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(
> Action.scala:110)
> at scala.Option.map(Option.scala:146)
> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(Ba
> tchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(
> BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
> 1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$
> 1.apply(BatchingExecutor.scala:91)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingEx
> ecutor.scala:90)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.
> exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
> ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPoo
> l.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinW
> orkerThread.java:107)
>
>

Re: objects properties as inverse and at the same time disjoint

Posted by Lorenz Buehmann <bu...@informatik.uni-leipzig.de>.
1. Are you sure that you're asking on the right mailing list?

2. HermiT is generating which error? You forgot to put it into the mail
- note, images are not supported

3. That is not an error in OWL 2 DL, but as we can't see the "error" and
don't know whether you're not having other axioms in your ontology that
might violate the global restrictions in OWL 2 DL [1], who knows?

4. OOPS is not an OWL DL reasoner, but a pitfall scanner with a catalog
of common pitfalls [2] in ontology modeling. This tool would probably
fail to determine OWL 2 DL limitations on properties as mentioned in [1]

5. So you have Pellet + TrOWL vs HermiT -> 2 vs 1 -> would this be
enough evidence for you?

6. You could try to contact Birte Glimm (principle developer/maintainer
of HermiT) and ask whether this is a bug or the other reasoners are wrong


[1]
https://www.w3.org/TR/owl2-syntax/#Global_Restrictions_on_Axioms_in_OWL_2_DL

[2] http://oops.linkeddata.es/catalogue.jsp


On 01.12.2017 21:40, Manuel Enrique Puebla Martinez wrote:
>
> Hello :
>
>   Is it an error to define two properties of objects as inverse to each other and at the same time disjunct?
>
> The reasoner HermiT 1.3.8.3 is generating the following error and I am suspecting that is why:
>
> However, other reasoners and evaluative tools such as OOPS, Pellet and TrOWL do not report this error.
>
> Best regards, MAnuel Puebla.
> La @universidad_uci es Fidel: 15 años conectados al futuro... conectados a la Revolución
> 2002-2017

objects properties as inverse and at the same time disjoint

Posted by Manuel Enrique Puebla Martinez <mp...@uci.cu>.

Hello :

  Is it an error to define two properties of objects as inverse to each other and at the same time disjunct?

The reasoner HermiT 1.3.8.3 is generating the following error and I am suspecting that is why:

However, other reasoners and evaluative tools such as OOPS, Pellet and TrOWL do not report this error.

Best regards, MAnuel Puebla.
La @universidad_uci es Fidel: 15 años conectados al futuro... conectados a la Revolución
2002-2017

Re: Slow Lucene query

Posted by Andy Seaborne <an...@apache.org>.

On 01/12/17 11:49, Jean-Marc Vanel wrote:
> Hi
> 
> The time for a SPARQL + Lucene query is more than 1 mn.
> It used to be around one second, and then the database grew.
> Here is a typical query run for a lookup service similar to dbpedia lookup:
> 
> PREFIX text: <http://jena.apache.org/text#>
> PREFIX form: <
> http://raw.githubusercontent.com/jmvanel/semantic_forms/master/vocabulary/forms.owl.ttl#
>>
> SELECT DISTINCT ?thing ?COUNT WHERE {
>    graph ?g {
>      ?thing text:query ( 'Jean-Marc' ) .
>    }

This is going to loop on each graph and make a text:query for each one.

Is that what you intended?

Remove the "graph ?g {".

(and then remove the DISTINCT)

>    graph ?g1 {
>      ?thing a ?CLASS .

Unnecessary?

>    }
>    OPTIONAL {
>     graph ?grCount {
>      ?thing form:linksCount ?COUNT.
>    } }
> } ORDER BY DESC(?COUNT)
> LIMIT 10
> 
> Here is a simpler query that is also slow:
> 
> PREFIX text: <http://jena.apache.org/text#>
> SELECT DISTINCT ?thing ?COUNT WHERE {
>    graph ?g {
>      ?thing text:query ( 'Jean-Marc' ) .
>    }
> } ORDER BY DESC(?COUNT)
> LIMIT 10
> 
> Run it with:
> time wget -O semantic-forms.cc_select-ui.txt
> http://semantic-forms.cc:9112/select-ui?query=PREFIX+text%3A+%3Chttp%3A%2F%2Fjena.apache.org%2Ftext%23%3E+%0D%0ASELECT+DISTINCT+%3Fthing+%3FCOUNT+WHERE+%7B%0D%0A++graph+%3Fg+%7B%0D%0A++++%3Fthing+text%3Aquery+%28+%27Jean-Marc%27+%29+.%0D%0A++%7D%0D%0A%7D%0D%0AORDER+BY+DESC%28%3FCOUNT%29%0D%0ALIMIT+10
> 
> Or, if you want to use YasGUI , the endpoint is
> http://semantic-forms.cc:9112/sparql
> 
> *Statistics on the database*
> 268 graphs and 588 864 triples.
> 
> # Count graphs and triples
> SELECT (COUNT(?s) AS ?trc) (COUNT(?GR) AS ?grc)
>      WHERE {
>    { GRAPH ?GR { } }
>    UNION
>    { GRAPH ?GR1 { ?s ?p ?o . } }
> }
> 
> Result: 2 rows
> "grc" "trc"
> "268"^^http://www.w3.org/2001/XMLSchema#integer "588864"^^
> http://www.w3.org/2001/XMLSchema#integer
> 
> (I'm not sure this the right way to count, but it gives figures :) )
> 
> You can reproduce the query with this UI :
> 
> http://semantic-forms.cc:9112/select-ui?query=%23+Count+graphs%0D%0ASELECT+%28COUNT%28%3Fs%29+AS+%3Ftrc%29+%28COUNT%28%3FGR%29+AS+%3Fgrc%29%0D%0A++++WHERE+%7B%0D%0A+++++%7B+GRAPH+%3FGR+%7B+%7D+%7D%0D%0A++UNION%0D%0A++++%7B+GRAPH+%3FGR1+%7B+%3Fs+%3Fp+%3Fo+.+%7D+%7D%0D%0A%7D
> 
> This is using Jena 3.5.0. with TDB 1 .
> Here is a stack made with kill -3 when the app. was working hard;
> I put in bold a suspect line.
> 
> "application-akka.actor.default-dispatcher-1351" #2683 prio=5 os_prio=0
> tid=0x00007f07a801c000 nid=0x9b7 runnable [0x00007f06f25ab000]
>     java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
> at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
> at
> org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194)
> at org.apache.lucene.util.fst.FST.<init>(FST.java:327)
> at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
> at
> org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldReader.java:91)
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:234)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:445)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:112)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:77)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
> at
> org.apache.jena.query.text.TextIndexLucene.query(TextIndexLucene.java:370)
> at org.apache.jena.query.text.TextQueryPF.performQuery(TextQueryPF.java:290)
> at
> org.apache.jena.query.text.TextQueryPF.lambda$query$1(TextQueryPF.java:267)
> at
> org.apache.jena.query.text.TextQueryPF$$Lambda$66/2108167189.call(Unknown
> Source)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5065)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
> * - locked <0x00000000ef9d34f8> (a
> org.apache.jena.ext.com.google.common.cache.LocalCache$StrongAccessEntry)*
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache.get(LocalCache.java:4154)
> at
> org.apache.jena.ext.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5060)
> at org.apache.jena.atlas.lib.cache.CacheGuava.getOrFill(CacheGuava.java:58)
> at org.apache.jena.query.text.TextQueryPF.query(TextQueryPF.java:267)
> at
> org.apache.jena.query.text.TextQueryPF.variableSubject(TextQueryPF.java:227)
> at org.apache.jena.query.text.TextQueryPF.exec(TextQueryPF.java:196)
> at
> org.apache.jena.sparql.pfunction.PropertyFunctionBase$RepeatApplyIteratorPF.nextStage(PropertyFunctionBase.java:106)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.makeNextStage(QueryIterRepeatApply.java:108)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:65)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterProcedure.hasNextBinding(QueryIterProcedure.java:73)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterProcessBinding.hasNextBinding(QueryIterProcessBinding.java:66)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.main.iterator.QueryIterGraph$QueryIterGraphInner.hasNextBinding(QueryIterGraph.java:121)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:74)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
> at
> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:45)
> at
> org.apache.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195)
> at org.apache.jena.atlas.iterator.Iter$2.hasNext(Iter.java:265)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:53)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.makeNextStage(QueryIterRepeatApply.java:101)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:65)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasNextBinding(QueryIterConvert.java:58)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterTopN$1.initializeIterator(QueryIterTopN.java:98)
> at
> org.apache.jena.atlas.iterator.IteratorDelayedInitialization.init(IteratorDelayedInitialization.java:40)
> at
> org.apache.jena.atlas.iterator.IteratorDelayedInitialization.hasNext(IteratorDelayedInitialization.java:50)
> at
> org.apache.jena.sparql.engine.iterator.QueryIterPlainWrapper.hasNextBinding(QueryIterPlainWrapper.java:53)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
> at
> org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
> at
> org.apache.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:74)
> at
> org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNext(ResultSetCheckCondition.java:55)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at scala.collection.Iterator$class.toStream(Iterator.scala:1320)
> at scala.collection.AbstractIterator.toStream(Iterator.scala:1334)
> at
> scala.collection.TraversableOnce$class.toIterable(TraversableOnce.scala:296)
> at scala.collection.AbstractIterator.toIterable(Iterator.scala:1334)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparqlSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:382)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$sparqlSelectQueryVariablesNT$1.apply(SPARQLHelpers.scala:370)
> at deductions.runtime.utils.Timer$class.time(Timer.scala:18)
> at controllers.Application$.time(Application.scala:8)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSelectQueryVariablesNT(SPARQLHelpers.scala:370)
> at
> controllers.Application$.sparqlSelectQueryVariablesNT(Application.scala:8)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.apply(SPARQLHelpers.scala:361)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$$anonfun$8.apply(SPARQLHelpers.scala:361)
> at
> org.w3.banana.jena.JenaDatasetStore$$anonfun$r$1.apply(JenaDatasetStore.scala:17)
> at scala.util.Try$.apply(Try.scala:192)
> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:14)
> at org.w3.banana.jena.JenaDatasetStore.r(JenaDatasetStore.scala:10)
> at
> deductions.runtime.sparql_cache.SPARQLHelpers$class.sparqlSelectQueryVariables(SPARQLHelpers.scala:359)
> at controllers.Application$.sparqlSelectQueryVariables(Application.scala:8)
> at
> deductions.runtime.services.Lookup$class.searchStringOrClass(Lookup.scala:76)
> at deductions.runtime.services.Lookup$class.lookup(Lookup.scala:44)
> at controllers.Application$.lookup(Application.scala:8)
> at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:199)
> at controllers.Services$$anonfun$lookupService$1.apply(Services.scala:193)
> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
> at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
> at play.api.mvc.Action$.invokeBlock(Action.scala:498)
> at play.api.mvc.Action$.invokeBlock(Action.scala:495)
> at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
> at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:111)
> at
> play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:110)
> at scala.Option.map(Option.scala:146)
> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
> at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
> at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>