You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Mark Wharton <ma...@iotic-labs.com> on 2015/12/18 16:13:21 UTC
Query with spatial *and* text searches.
Hi Jena users.
I'm having performance problems with a query that uses text and location
search
The query is roughly this:
SELECT ?score ?ent
WHERE {
?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.01807880401611 'km') .
(?ent ?score) text:query ('environment' 'lang:en') .
?ent rdf:type iotic:Entity .
}
There are about 450 entities in that radius
There are about 2200 entities with environment in their rdfs:comment
The query takes 5 seconds.
I've tried this:
Commenting out the text predicate the query takes 97 ms
Commenting out the spatial predicate the query takes 438 ms
Swapping the spatial and text predicates it takes 15 seconds
My question is this... It looks like the query is separately getting
the results of the first two predicates and merging (somehow) to find
the intersection. Is there a formulation which will intersect the two
sets faster?
Many TIAs,
Mark
--
Technology Lead, Iotic Labs
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
Re: Query with spatial *and* text searches.
Posted by Andy Seaborne <an...@apache.org>.
On 27/12/15 20:28, Mark Wharton wrote:
> Hi Andy
>
> Thanks for your help. I've integrated those changes into the main
> search query and it's about 10x faster.
>
> Just one last question... What does the OFFSET 0 bit do in the query -
> force it to use the spatial/text index in preference to any other?
It stops the optimizer flattening the query. Using that "idiom" (AKA a
bit of a hack) and causes the execution to do an hash join between two
index accesses.
Otherwise, the optimizer will flatten out the query and feed the results
of the first spatial:nearby in to the text:query one by one.
Andy
>
> Mark
>
> Technology Lead, Iotic Labs
> +44 7973 674404
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 24/12/15 05:38, Mark Wharton wrote:
>> Hi Andy.
>>
>> That's cracked it. I was wondering about the sub-select route, but
>> wasn't sure how to code the intersection part. I just tweaked it to
>> return the score from the text query
>>
>> Your formulation
>> 200 OK (231 ms)
>>
>> That's 200 OK by me...
>>
>> Enjoy the holidays
>>
>> Mark
>>
>> Technology Lead, Iotic Labs
>> +44 7973 674404
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 23/12/15 17:03, Andy Seaborne wrote:
>>> Hi Mark,
>>>
>>> Tricky.
>>>
>>> There isn't a good way to turn off or modify optimization for parts of a
>>> query without affecting the whole query. Jena 3.0.1 had a combination
>>> of changes - hash join but also stronger flattening queries into the
>>> form you don't want for the first part.
>>>
>>> The best I have come up with is:
>>> (no special flags needed)
>>>
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { SELECT ?ent { ?ent spatial:nearby "ABC" . } OFFSET 0 }
>>> { SELECT ?ent { ?ent text:query "DEF" . } OFFSET 0 }
>>> ... rest of query ...
>>>
>>> }
>>>
>>> i.e.
>>>
>>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>>> WHERE {
>>> { SELECT ?ent {
>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.8018078804016'km') .
>>> } OFFSET 0 }
>>> { SELECT ?ent {
>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>> } OFFSET 0 }
>>>
>>> ?ent rdf:type iotic:Entity
>>>
>>> OPTIONAL {
>>> ?ent rdfs:label ?entLabel .
>>> FILTER langMatches( lang(?entLabel), 'en' ) .
>>> }
>>>
>>> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>>> ?ent iotic:Advertises ?point .
>>> ?point rdf:type iotic:Point .
>>> ?point iotic:PointType ?pointType .
>>>
>>> OPTIONAL {
>>> ?point rdfs:label ?pointLabel .
>>> FILTER langMatches( lang(?pointLabel), 'en' ) .
>>> }
>>> }
>>>
>>>
>>> On 23/12/15 11:03, Mark Wharton wrote:
>>>> Hi Andy.
>>>>
>>>> More experiments this morning. I originally only send you a small part
>>>> of a larger query just to expose the problem in its simplest form. And
>>>> your switches work well in that case (i.e. first formulation below
>>>> *with* the comments.)
>>>>
>>>> But... There's a problem when using the switches in that the rest of the
>>>> query wants to get the rdfs:label and various other properties. This
>>>> destroys the performance gains.
>>>>
>>>> I've tried "yours" and "mine" with and without the switches and then the
>>>> separate parts on their own to see how that goes.
>>>>
>>>> 1) "yours"
>>>> ==========
>>>> This formulation (with the switches and comments in place) - 384 ms
>>>>
>>>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>>>> WHERE {
>>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>> 70.8018078804016'km') }
>>>> { (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
>>>> {?ent rdf:type iotic:Entity} }
>>>>
>>>> # OPTIONAL {
>>>> # ?ent rdfs:label ?entLabel .
>>>> # FILTER langMatches( lang(?entLabel), 'en' ) .
>>>> # }
>>>> #
>>>> # OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>>>> # ?ent iotic:Advertises ?point .
>>>> # ?point rdf:type iotic:Point .
>>>> # ?point iotic:PointType ?pointType .
>>>> #
>>>> # OPTIONAL {
>>>> # ?point rdfs:label ?pointLabel .
>>>> # FILTER langMatches( lang(?pointLabel), 'en' ) .
>>>> # }
>>>>
>>>> }
>>>>
>>>> Uncomment the lines and the performance drops to - 7.165 ms
>>>>
>>>> 2) "mine"
>>>> =========
>>>> The below formulation with the switches in place 11.221 secs
>>>> The below without the switches. 5.371 secs
>>>>
>>>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>>>> WHERE {
>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>> 70.8018078804016'km') .
>>>> (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
>>>> {?ent rdf:type iotic:Entity} .
>>>>
>>>> OPTIONAL {
>>>> ?ent rdfs:label ?entLabel .
>>>> FILTER langMatches( lang(?entLabel), 'en' ) .
>>>> }
>>>>
>>>> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>>>> ?ent iotic:Advertises ?point .
>>>> ?point rdf:type iotic:Point .
>>>> ?point iotic:PointType ?pointType .
>>>>
>>>> OPTIONAL {
>>>> ?point rdfs:label ?pointLabel .
>>>> FILTER langMatches( lang(?pointLabel), 'en' ) .
>>>> }
>>>>
>>>> }
>>>>
>>>> 3) Separately
>>>> ==============
>>>> Completely on their own:
>>>> ========================
>>>> i.e. just the ?ent spatial:nearby line
>>>> the spatial query on its own takes 50 ms
>>>> i.e just the text:query line
>>>> and the text on its own takes 258 ms
>>>>
>>>> With the OPTIONAL {} and other properties
>>>> =========================================
>>>> Spatial and other properties 135 ms
>>>> Text and other properties 854 ms
>>>>
>>>> Again, repeated thanks for you help.
>>>>
>>>> Mark
>>>>
>>>> Technology Lead, Iotic Labs
>>>> mark.wharton@iotic-labs.com
>>>> https://www.iotic-labs.com
>>>>
>>>> On 22/12/15 17:22, Andy Seaborne wrote:
>>>>> Mark,
>>>>>
>>>>> Thanks for the experiment results.
>>>>>
>>>>> On 22/12/15 15:47, Mark Wharton wrote:
>>>>>> Query below run without Andy's switches.
>>>>>> INFO [5] 200 OK (4.985 s)
>>>>>>
>>>>>> Query below run with Andy's switches.
>>>>>> INFO [1] 200 OK (840 ms)
>>>>>>
>>>>>> Them's some magic switches. Thanks, Andy.
>>>>>>
>>>>>> Do they have any impact (negative or positive) on any other SPARQL
>>>>>> operations? I'm only curious as you've solved our main problem in that
>>>>>> our "search" query was very slow. There's nowhere else that uses the
>>>>>> text and spatial indexes for retrieval.
>>>>>
>>>>> This depends on any internal change in the latest release (Jena 3.0.1,
>>>>> Fuseki 2.3.1). Prior to that it will not make the same difference.
>>>>> Specially, unoptimized joins are now hash joins.
>>>>>
>>>>> But that is not a good choice for the "?ent rdf:type iotic:Entity"
>>>>> triple pattern. The system can't distinguish different cases involving
>>>>> external indexes as it knows not very much about the index details.
>>>>>
>>>>> Adding
>>>>>
>>>>> FILTER EXISTS { ?ext rdf:type iotic:Entity }
>>>>>
>>>>> might work because the triple pattern is really a check, not a match
>>>>> setting a variable.
>>>>>
>>>>> A plain "?ent rdf:type iotic:Entity" will retrieve all things of that
>>>>> class regardless of spatial and text query when those optimization
>>>>> are off.
>>>>>
>>>>> Andy
>>>>>
>>>>>>
>>>>>> Many thanks for this help so close to the holiday season. Happy
>>>>>> holidays to you all at Jena - keep up the good work.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>> Technology Lead, Iotic Labs
>>>>>> +44 7973 674404
>>>>>> mark.wharton@iotic-labs.com
>>>>>> https://www.iotic-labs.com
>>>>>>
>>>>>> On 22/12/15 11:49, Andy Seaborne wrote:
>>>>>>> Mark - here is another way.
>>>>>>>
>>>>>>> This query:
>>>>>>>
>>>>>>> SELECT ?score ?ent
>>>>>>> WHERE {
>>>>>>> { ?ent spatial:nearby ( .... ) }
>>>>>>> { ?ent text:query ( ..... ) }
>>>>>>> # No ?ent rdf:type iotic:Entity .
>>>>>>> # This focuses the query on the presenting issue.
>>>>>>> }
>>>>>>>
>>>>>>> and then run Fuseki with the following flags:
>>>>>>>
>>>>>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>>>>>>>
>>>>>>> for however you are running the server.
>>>>>>>
>>>>>>> You need both --set
>>>>>>>
>>>>>>> The service script will not do this very easily - if environment
>>>>>>> variable FUSEKI_ARGS is set it might do. Untested.
>>>>>>>
>>>>>>> It is easier to run the server standalone:
>>>>>>>
>>>>>>> (Linux, Mac)
>>>>>>>
>>>>>>> The "fuseki-server" script should pass these in:
>>>>>>>
>>>>>>> fuseki-server \
>>>>>>> --set arq:optIndexJoinStrategy=false --set
>>>>>>> arq:optMergeBGPs=false \
>>>>>>> .. other args ..
>>>>>>>
>>>>>>> (Windows or any platform)
>>>>>>>
>>>>>>> You can call the server java code directly: all one line:
>>>>>>>
>>>>>>>
>>>>>>> java -Xmx1200M -jar fuseki-server.jar --set
>>>>>>> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
>>>>>>> args ..
>>>>>>>
>>>>>>> you'll need to put the full path name of fuseki-server.jar
>>>>>>>
>>>>>>> Sorry - I don't have your setup to test this fully. I did make sure
>>>>>>> that
>>>>>>> the reworked query does lead to an execution plan that is different
>>>>>>> and
>>>>>>> should yield some information about the situation.
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> On 22/12/15 09:50, Andy Seaborne wrote:
>>>>>>>> On 22/12/15 07:06, Mark Wharton wrote:
>>>>>>>>> Ah, wheels within wheels.
>>>>>>>>>
>>>>>>>>> The formulation with the filter in it is fine, except that if you
>>>>>>>>> want
>>>>>>>>> to search for more than one word or you match in label and comment
>>>>>>>>> then
>>>>>>>>> the UNION formulation returns you duplicate rows. This isn't a
>>>>>>>>> problem
>>>>>>>>> with the Lucene search which is why (I now remember) I used it in
>>>>>>>>> the
>>>>>>>>> first place.
>>>>>>>>>
>>>>>>>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>>>>>>>> release at 2.3.0. Is there a way to find out?
>>>>>>>>
>>>>>>>> 3.0.0
>>>>>>>>
>>>>>>>> Many of the java commands support --version and the fuseki- server
>>>>>>>> jar
>>>>>>>> is an all-in-one jar:
>>>>>>>>
>>>>>>>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>>>>>>>
>>>>>>>>> What's the status on the JENA-999 and JENA-1093 issues? I see
>>>>>>>>> there's
>>>>>>>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>>>>>>>> comment seems encouraging.
>>>>>>>>>
>>>>>>>>> I don't want to adopt a single version as I'll be stuck forever
>>>>>>>>> patching
>>>>>>>>> back and forward and it will break eventually.
>>>>>>>>>
>>>>>>>>> Many thanks for your continued help.
>>>>>>>>
>>>>>>>> JENA-999 may sort of help but I'm not that positive because each ?ent
>>>>>>>> from the first part will be different going into the second part. It
>>>>>>>> looks to me as if it is the overhead of going out to Lucene. (This is
>>>>>>>> Lucene right? not Solr?)
>>>>>>>>
>>>>>>>> The ideal is some super compilation of the text:query and spatial
>>>>>>>> query
>>>>>>>> into one big Lucene query.
>>>>>>>>
>>>>>>>> What would also be good, which is stop the general optimizer (this is
>>>>>>>> nothing to do with TDB) using an index join. Except that is the
>>>>>>>> better
>>>>>>>> choice for the rdf:type. This is what the addition {} were trying
>>>>>>>> for
>>>>>>>> except the optimizer outsmarted
>>>>>>>>
>>>>>>>> SELECT ?score ?ent
>>>>>>>> WHERE {
>>>>>>>> ?ent spatial:nearby( ...) .
>>>>>>>> (?ent ?score) text:query (...) .
>>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Mark - can you ask the query from Java? If so,
>>>>>>>>
>>>>>>>> Add "Optimize.noOptimizer(); " before executing the query. I can't
>>>>>>>> see
>>>>>>>> a way to do that from setting the environment for Fuseki.
>>>>>>>>
>>>>>>>> Or (the effect on time of this is version specific and whether it
>>>>>>>> does
>>>>>>>> anything useful is a big "maybe") you could try this:
>>>>>>>>
>>>>>>>> SELECT ?score ?ent
>>>>>>>> WHERE {
>>>>>>>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>>>>>>>> { OPTIONAL { ?ent text:query "DEF" } }
>>>>>>>> }
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi Andy
Thanks for your help. I've integrated those changes into the main
search query and it's about 10x faster.
Just one last question... What does the OFFSET 0 bit do in the query -
force it to use the spatial/text index in preference to any other?
Mark
Technology Lead, Iotic Labs
+44 7973 674404
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 24/12/15 05:38, Mark Wharton wrote:
> Hi Andy.
>
> That's cracked it. I was wondering about the sub-select route, but
> wasn't sure how to code the intersection part. I just tweaked it to
> return the score from the text query
>
> Your formulation
> 200 OK (231 ms)
>
> That's 200 OK by me...
>
> Enjoy the holidays
>
> Mark
>
> Technology Lead, Iotic Labs
> +44 7973 674404
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 23/12/15 17:03, Andy Seaborne wrote:
>> Hi Mark,
>>
>> Tricky.
>>
>> There isn't a good way to turn off or modify optimization for parts of a
>> query without affecting the whole query. Jena 3.0.1 had a combination
>> of changes - hash join but also stronger flattening queries into the
>> form you don't want for the first part.
>>
>> The best I have come up with is:
>> (no special flags needed)
>>
>>
>> SELECT ?score ?ent
>> WHERE {
>> { SELECT ?ent { ?ent spatial:nearby "ABC" . } OFFSET 0 }
>> { SELECT ?ent { ?ent text:query "DEF" . } OFFSET 0 }
>> ... rest of query ...
>>
>> }
>>
>> i.e.
>>
>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>> WHERE {
>> { SELECT ?ent {
>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.8018078804016'km') .
>> } OFFSET 0 }
>> { SELECT ?ent {
>> (?ent ?score) text:query ('environment' 'lang:en') .
>> } OFFSET 0 }
>>
>> ?ent rdf:type iotic:Entity
>>
>> OPTIONAL {
>> ?ent rdfs:label ?entLabel .
>> FILTER langMatches( lang(?entLabel), 'en' ) .
>> }
>>
>> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>> ?ent iotic:Advertises ?point .
>> ?point rdf:type iotic:Point .
>> ?point iotic:PointType ?pointType .
>>
>> OPTIONAL {
>> ?point rdfs:label ?pointLabel .
>> FILTER langMatches( lang(?pointLabel), 'en' ) .
>> }
>> }
>>
>>
>> On 23/12/15 11:03, Mark Wharton wrote:
>>> Hi Andy.
>>>
>>> More experiments this morning. I originally only send you a small part
>>> of a larger query just to expose the problem in its simplest form. And
>>> your switches work well in that case (i.e. first formulation below
>>> *with* the comments.)
>>>
>>> But... There's a problem when using the switches in that the rest of the
>>> query wants to get the rdfs:label and various other properties. This
>>> destroys the performance gains.
>>>
>>> I've tried "yours" and "mine" with and without the switches and then the
>>> separate parts on their own to see how that goes.
>>>
>>> 1) "yours"
>>> ==========
>>> This formulation (with the switches and comments in place) - 384 ms
>>>
>>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>>> WHERE {
>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.8018078804016'km') }
>>> { (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
>>> {?ent rdf:type iotic:Entity} }
>>>
>>> # OPTIONAL {
>>> # ?ent rdfs:label ?entLabel .
>>> # FILTER langMatches( lang(?entLabel), 'en' ) .
>>> # }
>>> #
>>> # OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>>> # ?ent iotic:Advertises ?point .
>>> # ?point rdf:type iotic:Point .
>>> # ?point iotic:PointType ?pointType .
>>> #
>>> # OPTIONAL {
>>> # ?point rdfs:label ?pointLabel .
>>> # FILTER langMatches( lang(?pointLabel), 'en' ) .
>>> # }
>>>
>>> }
>>>
>>> Uncomment the lines and the performance drops to - 7.165 ms
>>>
>>> 2) "mine"
>>> =========
>>> The below formulation with the switches in place 11.221 secs
>>> The below without the switches. 5.371 secs
>>>
>>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>>> WHERE {
>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.8018078804016'km') .
>>> (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
>>> {?ent rdf:type iotic:Entity} .
>>>
>>> OPTIONAL {
>>> ?ent rdfs:label ?entLabel .
>>> FILTER langMatches( lang(?entLabel), 'en' ) .
>>> }
>>>
>>> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>>> ?ent iotic:Advertises ?point .
>>> ?point rdf:type iotic:Point .
>>> ?point iotic:PointType ?pointType .
>>>
>>> OPTIONAL {
>>> ?point rdfs:label ?pointLabel .
>>> FILTER langMatches( lang(?pointLabel), 'en' ) .
>>> }
>>>
>>> }
>>>
>>> 3) Separately
>>> ==============
>>> Completely on their own:
>>> ========================
>>> i.e. just the ?ent spatial:nearby line
>>> the spatial query on its own takes 50 ms
>>> i.e just the text:query line
>>> and the text on its own takes 258 ms
>>>
>>> With the OPTIONAL {} and other properties
>>> =========================================
>>> Spatial and other properties 135 ms
>>> Text and other properties 854 ms
>>>
>>> Again, repeated thanks for you help.
>>>
>>> Mark
>>>
>>> Technology Lead, Iotic Labs
>>> mark.wharton@iotic-labs.com
>>> https://www.iotic-labs.com
>>>
>>> On 22/12/15 17:22, Andy Seaborne wrote:
>>>> Mark,
>>>>
>>>> Thanks for the experiment results.
>>>>
>>>> On 22/12/15 15:47, Mark Wharton wrote:
>>>>> Query below run without Andy's switches.
>>>>> INFO [5] 200 OK (4.985 s)
>>>>>
>>>>> Query below run with Andy's switches.
>>>>> INFO [1] 200 OK (840 ms)
>>>>>
>>>>> Them's some magic switches. Thanks, Andy.
>>>>>
>>>>> Do they have any impact (negative or positive) on any other SPARQL
>>>>> operations? I'm only curious as you've solved our main problem in that
>>>>> our "search" query was very slow. There's nowhere else that uses the
>>>>> text and spatial indexes for retrieval.
>>>>
>>>> This depends on any internal change in the latest release (Jena 3.0.1,
>>>> Fuseki 2.3.1). Prior to that it will not make the same difference.
>>>> Specially, unoptimized joins are now hash joins.
>>>>
>>>> But that is not a good choice for the "?ent rdf:type iotic:Entity"
>>>> triple pattern. The system can't distinguish different cases involving
>>>> external indexes as it knows not very much about the index details.
>>>>
>>>> Adding
>>>>
>>>> FILTER EXISTS { ?ext rdf:type iotic:Entity }
>>>>
>>>> might work because the triple pattern is really a check, not a match
>>>> setting a variable.
>>>>
>>>> A plain "?ent rdf:type iotic:Entity" will retrieve all things of that
>>>> class regardless of spatial and text query when those optimization
>>>> are off.
>>>>
>>>> Andy
>>>>
>>>>>
>>>>> Many thanks for this help so close to the holiday season. Happy
>>>>> holidays to you all at Jena - keep up the good work.
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> Technology Lead, Iotic Labs
>>>>> +44 7973 674404
>>>>> mark.wharton@iotic-labs.com
>>>>> https://www.iotic-labs.com
>>>>>
>>>>> On 22/12/15 11:49, Andy Seaborne wrote:
>>>>>> Mark - here is another way.
>>>>>>
>>>>>> This query:
>>>>>>
>>>>>> SELECT ?score ?ent
>>>>>> WHERE {
>>>>>> { ?ent spatial:nearby ( .... ) }
>>>>>> { ?ent text:query ( ..... ) }
>>>>>> # No ?ent rdf:type iotic:Entity .
>>>>>> # This focuses the query on the presenting issue.
>>>>>> }
>>>>>>
>>>>>> and then run Fuseki with the following flags:
>>>>>>
>>>>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>>>>>>
>>>>>> for however you are running the server.
>>>>>>
>>>>>> You need both --set
>>>>>>
>>>>>> The service script will not do this very easily - if environment
>>>>>> variable FUSEKI_ARGS is set it might do. Untested.
>>>>>>
>>>>>> It is easier to run the server standalone:
>>>>>>
>>>>>> (Linux, Mac)
>>>>>>
>>>>>> The "fuseki-server" script should pass these in:
>>>>>>
>>>>>> fuseki-server \
>>>>>> --set arq:optIndexJoinStrategy=false --set
>>>>>> arq:optMergeBGPs=false \
>>>>>> .. other args ..
>>>>>>
>>>>>> (Windows or any platform)
>>>>>>
>>>>>> You can call the server java code directly: all one line:
>>>>>>
>>>>>>
>>>>>> java -Xmx1200M -jar fuseki-server.jar --set
>>>>>> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
>>>>>> args ..
>>>>>>
>>>>>> you'll need to put the full path name of fuseki-server.jar
>>>>>>
>>>>>> Sorry - I don't have your setup to test this fully. I did make sure
>>>>>> that
>>>>>> the reworked query does lead to an execution plan that is different
>>>>>> and
>>>>>> should yield some information about the situation.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> On 22/12/15 09:50, Andy Seaborne wrote:
>>>>>>> On 22/12/15 07:06, Mark Wharton wrote:
>>>>>>>> Ah, wheels within wheels.
>>>>>>>>
>>>>>>>> The formulation with the filter in it is fine, except that if you
>>>>>>>> want
>>>>>>>> to search for more than one word or you match in label and comment
>>>>>>>> then
>>>>>>>> the UNION formulation returns you duplicate rows. This isn't a
>>>>>>>> problem
>>>>>>>> with the Lucene search which is why (I now remember) I used it in
>>>>>>>> the
>>>>>>>> first place.
>>>>>>>>
>>>>>>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>>>>>>> release at 2.3.0. Is there a way to find out?
>>>>>>>
>>>>>>> 3.0.0
>>>>>>>
>>>>>>> Many of the java commands support --version and the fuseki- server
>>>>>>> jar
>>>>>>> is an all-in-one jar:
>>>>>>>
>>>>>>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>>>>>>
>>>>>>>> What's the status on the JENA-999 and JENA-1093 issues? I see
>>>>>>>> there's
>>>>>>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>>>>>>> comment seems encouraging.
>>>>>>>>
>>>>>>>> I don't want to adopt a single version as I'll be stuck forever
>>>>>>>> patching
>>>>>>>> back and forward and it will break eventually.
>>>>>>>>
>>>>>>>> Many thanks for your continued help.
>>>>>>>
>>>>>>> JENA-999 may sort of help but I'm not that positive because each ?ent
>>>>>>> from the first part will be different going into the second part. It
>>>>>>> looks to me as if it is the overhead of going out to Lucene. (This is
>>>>>>> Lucene right? not Solr?)
>>>>>>>
>>>>>>> The ideal is some super compilation of the text:query and spatial
>>>>>>> query
>>>>>>> into one big Lucene query.
>>>>>>>
>>>>>>> What would also be good, which is stop the general optimizer (this is
>>>>>>> nothing to do with TDB) using an index join. Except that is the
>>>>>>> better
>>>>>>> choice for the rdf:type. This is what the addition {} were trying
>>>>>>> for
>>>>>>> except the optimizer outsmarted
>>>>>>>
>>>>>>> SELECT ?score ?ent
>>>>>>> WHERE {
>>>>>>> ?ent spatial:nearby( ...) .
>>>>>>> (?ent ?score) text:query (...) .
>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Mark - can you ask the query from Java? If so,
>>>>>>>
>>>>>>> Add "Optimize.noOptimizer(); " before executing the query. I can't
>>>>>>> see
>>>>>>> a way to do that from setting the environment for Fuseki.
>>>>>>>
>>>>>>> Or (the effect on time of this is version specific and whether it
>>>>>>> does
>>>>>>> anything useful is a big "maybe") you could try this:
>>>>>>>
>>>>>>> SELECT ?score ?ent
>>>>>>> WHERE {
>>>>>>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>>>>>>> { OPTIONAL { ?ent text:query "DEF" } }
>>>>>>> }
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi Andy.
That's cracked it. I was wondering about the sub-select route, but
wasn't sure how to code the intersection part. I just tweaked it to
return the score from the text query
Your formulation
200 OK (231 ms)
That's 200 OK by me...
Enjoy the holidays
Mark
Technology Lead, Iotic Labs
+44 7973 674404
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 23/12/15 17:03, Andy Seaborne wrote:
> Hi Mark,
>
> Tricky.
>
> There isn't a good way to turn off or modify optimization for parts of a
> query without affecting the whole query. Jena 3.0.1 had a combination
> of changes - hash join but also stronger flattening queries into the
> form you don't want for the first part.
>
> The best I have come up with is:
> (no special flags needed)
>
>
> SELECT ?score ?ent
> WHERE {
> { SELECT ?ent { ?ent spatial:nearby "ABC" . } OFFSET 0 }
> { SELECT ?ent { ?ent text:query "DEF" . } OFFSET 0 }
> ... rest of query ...
>
> }
>
> i.e.
>
> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
> WHERE {
> { SELECT ?ent {
> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.8018078804016'km') .
> } OFFSET 0 }
> { SELECT ?ent {
> (?ent ?score) text:query ('environment' 'lang:en') .
> } OFFSET 0 }
>
> ?ent rdf:type iotic:Entity
>
> OPTIONAL {
> ?ent rdfs:label ?entLabel .
> FILTER langMatches( lang(?entLabel), 'en' ) .
> }
>
> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
> ?ent iotic:Advertises ?point .
> ?point rdf:type iotic:Point .
> ?point iotic:PointType ?pointType .
>
> OPTIONAL {
> ?point rdfs:label ?pointLabel .
> FILTER langMatches( lang(?pointLabel), 'en' ) .
> }
> }
>
>
> On 23/12/15 11:03, Mark Wharton wrote:
>> Hi Andy.
>>
>> More experiments this morning. I originally only send you a small part
>> of a larger query just to expose the problem in its simplest form. And
>> your switches work well in that case (i.e. first formulation below
>> *with* the comments.)
>>
>> But... There's a problem when using the switches in that the rest of the
>> query wants to get the rdfs:label and various other properties. This
>> destroys the performance gains.
>>
>> I've tried "yours" and "mine" with and without the switches and then the
>> separate parts on their own to see how that goes.
>>
>> 1) "yours"
>> ==========
>> This formulation (with the switches and comments in place) - 384 ms
>>
>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>> WHERE {
>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.8018078804016'km') }
>> { (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
>> {?ent rdf:type iotic:Entity} }
>>
>> # OPTIONAL {
>> # ?ent rdfs:label ?entLabel .
>> # FILTER langMatches( lang(?entLabel), 'en' ) .
>> # }
>> #
>> # OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>> # ?ent iotic:Advertises ?point .
>> # ?point rdf:type iotic:Point .
>> # ?point iotic:PointType ?pointType .
>> #
>> # OPTIONAL {
>> # ?point rdfs:label ?pointLabel .
>> # FILTER langMatches( lang(?pointLabel), 'en' ) .
>> # }
>>
>> }
>>
>> Uncomment the lines and the performance drops to - 7.165 ms
>>
>> 2) "mine"
>> =========
>> The below formulation with the switches in place 11.221 secs
>> The below without the switches. 5.371 secs
>>
>> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
>> WHERE {
>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.8018078804016'km') .
>> (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
>> {?ent rdf:type iotic:Entity} .
>>
>> OPTIONAL {
>> ?ent rdfs:label ?entLabel .
>> FILTER langMatches( lang(?entLabel), 'en' ) .
>> }
>>
>> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
>> ?ent iotic:Advertises ?point .
>> ?point rdf:type iotic:Point .
>> ?point iotic:PointType ?pointType .
>>
>> OPTIONAL {
>> ?point rdfs:label ?pointLabel .
>> FILTER langMatches( lang(?pointLabel), 'en' ) .
>> }
>>
>> }
>>
>> 3) Separately
>> ==============
>> Completely on their own:
>> ========================
>> i.e. just the ?ent spatial:nearby line
>> the spatial query on its own takes 50 ms
>> i.e just the text:query line
>> and the text on its own takes 258 ms
>>
>> With the OPTIONAL {} and other properties
>> =========================================
>> Spatial and other properties 135 ms
>> Text and other properties 854 ms
>>
>> Again, repeated thanks for you help.
>>
>> Mark
>>
>> Technology Lead, Iotic Labs
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 22/12/15 17:22, Andy Seaborne wrote:
>>> Mark,
>>>
>>> Thanks for the experiment results.
>>>
>>> On 22/12/15 15:47, Mark Wharton wrote:
>>>> Query below run without Andy's switches.
>>>> INFO [5] 200 OK (4.985 s)
>>>>
>>>> Query below run with Andy's switches.
>>>> INFO [1] 200 OK (840 ms)
>>>>
>>>> Them's some magic switches. Thanks, Andy.
>>>>
>>>> Do they have any impact (negative or positive) on any other SPARQL
>>>> operations? I'm only curious as you've solved our main problem in that
>>>> our "search" query was very slow. There's nowhere else that uses the
>>>> text and spatial indexes for retrieval.
>>>
>>> This depends on any internal change in the latest release (Jena 3.0.1,
>>> Fuseki 2.3.1). Prior to that it will not make the same difference.
>>> Specially, unoptimized joins are now hash joins.
>>>
>>> But that is not a good choice for the "?ent rdf:type iotic:Entity"
>>> triple pattern. The system can't distinguish different cases involving
>>> external indexes as it knows not very much about the index details.
>>>
>>> Adding
>>>
>>> FILTER EXISTS { ?ext rdf:type iotic:Entity }
>>>
>>> might work because the triple pattern is really a check, not a match
>>> setting a variable.
>>>
>>> A plain "?ent rdf:type iotic:Entity" will retrieve all things of that
>>> class regardless of spatial and text query when those optimization
>>> are off.
>>>
>>> Andy
>>>
>>>>
>>>> Many thanks for this help so close to the holiday season. Happy
>>>> holidays to you all at Jena - keep up the good work.
>>>>
>>>> Mark
>>>>
>>>>
>>>> Technology Lead, Iotic Labs
>>>> +44 7973 674404
>>>> mark.wharton@iotic-labs.com
>>>> https://www.iotic-labs.com
>>>>
>>>> On 22/12/15 11:49, Andy Seaborne wrote:
>>>>> Mark - here is another way.
>>>>>
>>>>> This query:
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> { ?ent spatial:nearby ( .... ) }
>>>>> { ?ent text:query ( ..... ) }
>>>>> # No ?ent rdf:type iotic:Entity .
>>>>> # This focuses the query on the presenting issue.
>>>>> }
>>>>>
>>>>> and then run Fuseki with the following flags:
>>>>>
>>>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>>>>>
>>>>> for however you are running the server.
>>>>>
>>>>> You need both --set
>>>>>
>>>>> The service script will not do this very easily - if environment
>>>>> variable FUSEKI_ARGS is set it might do. Untested.
>>>>>
>>>>> It is easier to run the server standalone:
>>>>>
>>>>> (Linux, Mac)
>>>>>
>>>>> The "fuseki-server" script should pass these in:
>>>>>
>>>>> fuseki-server \
>>>>> --set arq:optIndexJoinStrategy=false --set
>>>>> arq:optMergeBGPs=false \
>>>>> .. other args ..
>>>>>
>>>>> (Windows or any platform)
>>>>>
>>>>> You can call the server java code directly: all one line:
>>>>>
>>>>>
>>>>> java -Xmx1200M -jar fuseki-server.jar --set
>>>>> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
>>>>> args ..
>>>>>
>>>>> you'll need to put the full path name of fuseki-server.jar
>>>>>
>>>>> Sorry - I don't have your setup to test this fully. I did make sure
>>>>> that
>>>>> the reworked query does lead to an execution plan that is different
>>>>> and
>>>>> should yield some information about the situation.
>>>>>
>>>>> Andy
>>>>>
>>>>> On 22/12/15 09:50, Andy Seaborne wrote:
>>>>>> On 22/12/15 07:06, Mark Wharton wrote:
>>>>>>> Ah, wheels within wheels.
>>>>>>>
>>>>>>> The formulation with the filter in it is fine, except that if you
>>>>>>> want
>>>>>>> to search for more than one word or you match in label and comment
>>>>>>> then
>>>>>>> the UNION formulation returns you duplicate rows. This isn't a
>>>>>>> problem
>>>>>>> with the Lucene search which is why (I now remember) I used it in
>>>>>>> the
>>>>>>> first place.
>>>>>>>
>>>>>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>>>>>> release at 2.3.0. Is there a way to find out?
>>>>>>
>>>>>> 3.0.0
>>>>>>
>>>>>> Many of the java commands support --version and the fuseki- server
>>>>>> jar
>>>>>> is an all-in-one jar:
>>>>>>
>>>>>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>>>>>
>>>>>>> What's the status on the JENA-999 and JENA-1093 issues? I see
>>>>>>> there's
>>>>>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>>>>>> comment seems encouraging.
>>>>>>>
>>>>>>> I don't want to adopt a single version as I'll be stuck forever
>>>>>>> patching
>>>>>>> back and forward and it will break eventually.
>>>>>>>
>>>>>>> Many thanks for your continued help.
>>>>>>
>>>>>> JENA-999 may sort of help but I'm not that positive because each ?ent
>>>>>> from the first part will be different going into the second part. It
>>>>>> looks to me as if it is the overhead of going out to Lucene. (This is
>>>>>> Lucene right? not Solr?)
>>>>>>
>>>>>> The ideal is some super compilation of the text:query and spatial
>>>>>> query
>>>>>> into one big Lucene query.
>>>>>>
>>>>>> What would also be good, which is stop the general optimizer (this is
>>>>>> nothing to do with TDB) using an index join. Except that is the
>>>>>> better
>>>>>> choice for the rdf:type. This is what the addition {} were trying
>>>>>> for
>>>>>> except the optimizer outsmarted
>>>>>>
>>>>>> SELECT ?score ?ent
>>>>>> WHERE {
>>>>>> ?ent spatial:nearby( ...) .
>>>>>> (?ent ?score) text:query (...) .
>>>>>> ?ent rdf:type iotic:Entity .
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> Mark - can you ask the query from Java? If so,
>>>>>>
>>>>>> Add "Optimize.noOptimizer(); " before executing the query. I can't
>>>>>> see
>>>>>> a way to do that from setting the environment for Fuseki.
>>>>>>
>>>>>> Or (the effect on time of this is version specific and whether it
>>>>>> does
>>>>>> anything useful is a big "maybe") you could try this:
>>>>>>
>>>>>> SELECT ?score ?ent
>>>>>> WHERE {
>>>>>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>>>>>> { OPTIONAL { ?ent text:query "DEF" } }
>>>>>> }
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>
>>>>>
>>>
>
Re: Query with spatial *and* text searches.
Posted by Andy Seaborne <an...@apache.org>.
Hi Mark,
Tricky.
There isn't a good way to turn off or modify optimization for parts of a
query without affecting the whole query. Jena 3.0.1 had a combination
of changes - hash join but also stronger flattening queries into the
form you don't want for the first part.
The best I have come up with is:
(no special flags needed)
SELECT ?score ?ent
WHERE {
{ SELECT ?ent { ?ent spatial:nearby "ABC" . } OFFSET 0 }
{ SELECT ?ent { ?ent text:query "DEF" . } OFFSET 0 }
... rest of query ...
}
i.e.
SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
WHERE {
{ SELECT ?ent {
?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.8018078804016'km') .
} OFFSET 0 }
{ SELECT ?ent {
(?ent ?score) text:query ('environment' 'lang:en') .
} OFFSET 0 }
?ent rdf:type iotic:Entity
OPTIONAL {
?ent rdfs:label ?entLabel .
FILTER langMatches( lang(?entLabel), 'en' ) .
}
OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
?ent iotic:Advertises ?point .
?point rdf:type iotic:Point .
?point iotic:PointType ?pointType .
OPTIONAL {
?point rdfs:label ?pointLabel .
FILTER langMatches( lang(?pointLabel), 'en' ) .
}
}
On 23/12/15 11:03, Mark Wharton wrote:
> Hi Andy.
>
> More experiments this morning. I originally only send you a small part
> of a larger query just to expose the problem in its simplest form. And
> your switches work well in that case (i.e. first formulation below
> *with* the comments.)
>
> But... There's a problem when using the switches in that the rest of the
> query wants to get the rdfs:label and various other properties. This
> destroys the performance gains.
>
> I've tried "yours" and "mine" with and without the switches and then the
> separate parts on their own to see how that goes.
>
> 1) "yours"
> ==========
> This formulation (with the switches and comments in place) - 384 ms
>
> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
> WHERE {
> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.8018078804016'km') }
> { (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
> {?ent rdf:type iotic:Entity} }
>
> # OPTIONAL {
> # ?ent rdfs:label ?entLabel .
> # FILTER langMatches( lang(?entLabel), 'en' ) .
> # }
> #
> # OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
> # ?ent iotic:Advertises ?point .
> # ?point rdf:type iotic:Point .
> # ?point iotic:PointType ?pointType .
> #
> # OPTIONAL {
> # ?point rdfs:label ?pointLabel .
> # FILTER langMatches( lang(?pointLabel), 'en' ) .
> # }
>
> }
>
> Uncomment the lines and the performance drops to - 7.165 ms
>
> 2) "mine"
> =========
> The below formulation with the switches in place 11.221 secs
> The below without the switches. 5.371 secs
>
> SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
> WHERE {
> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.8018078804016'km') .
> (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
> {?ent rdf:type iotic:Entity} .
>
> OPTIONAL {
> ?ent rdfs:label ?entLabel .
> FILTER langMatches( lang(?entLabel), 'en' ) .
> }
>
> OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
> ?ent iotic:Advertises ?point .
> ?point rdf:type iotic:Point .
> ?point iotic:PointType ?pointType .
>
> OPTIONAL {
> ?point rdfs:label ?pointLabel .
> FILTER langMatches( lang(?pointLabel), 'en' ) .
> }
>
> }
>
> 3) Separately
> ==============
> Completely on their own:
> ========================
> i.e. just the ?ent spatial:nearby line
> the spatial query on its own takes 50 ms
> i.e just the text:query line
> and the text on its own takes 258 ms
>
> With the OPTIONAL {} and other properties
> =========================================
> Spatial and other properties 135 ms
> Text and other properties 854 ms
>
> Again, repeated thanks for you help.
>
> Mark
>
> Technology Lead, Iotic Labs
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 22/12/15 17:22, Andy Seaborne wrote:
>> Mark,
>>
>> Thanks for the experiment results.
>>
>> On 22/12/15 15:47, Mark Wharton wrote:
>>> Query below run without Andy's switches.
>>> INFO [5] 200 OK (4.985 s)
>>>
>>> Query below run with Andy's switches.
>>> INFO [1] 200 OK (840 ms)
>>>
>>> Them's some magic switches. Thanks, Andy.
>>>
>>> Do they have any impact (negative or positive) on any other SPARQL
>>> operations? I'm only curious as you've solved our main problem in that
>>> our "search" query was very slow. There's nowhere else that uses the
>>> text and spatial indexes for retrieval.
>>
>> This depends on any internal change in the latest release (Jena 3.0.1,
>> Fuseki 2.3.1). Prior to that it will not make the same difference.
>> Specially, unoptimized joins are now hash joins.
>>
>> But that is not a good choice for the "?ent rdf:type iotic:Entity"
>> triple pattern. The system can't distinguish different cases involving
>> external indexes as it knows not very much about the index details.
>>
>> Adding
>>
>> FILTER EXISTS { ?ext rdf:type iotic:Entity }
>>
>> might work because the triple pattern is really a check, not a match
>> setting a variable.
>>
>> A plain "?ent rdf:type iotic:Entity" will retrieve all things of that
>> class regardless of spatial and text query when those optimization are off.
>>
>> Andy
>>
>>>
>>> Many thanks for this help so close to the holiday season. Happy
>>> holidays to you all at Jena - keep up the good work.
>>>
>>> Mark
>>>
>>>
>>> Technology Lead, Iotic Labs
>>> +44 7973 674404
>>> mark.wharton@iotic-labs.com
>>> https://www.iotic-labs.com
>>>
>>> On 22/12/15 11:49, Andy Seaborne wrote:
>>>> Mark - here is another way.
>>>>
>>>> This query:
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> { ?ent spatial:nearby ( .... ) }
>>>> { ?ent text:query ( ..... ) }
>>>> # No ?ent rdf:type iotic:Entity .
>>>> # This focuses the query on the presenting issue.
>>>> }
>>>>
>>>> and then run Fuseki with the following flags:
>>>>
>>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>>>>
>>>> for however you are running the server.
>>>>
>>>> You need both --set
>>>>
>>>> The service script will not do this very easily - if environment
>>>> variable FUSEKI_ARGS is set it might do. Untested.
>>>>
>>>> It is easier to run the server standalone:
>>>>
>>>> (Linux, Mac)
>>>>
>>>> The "fuseki-server" script should pass these in:
>>>>
>>>> fuseki-server \
>>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false \
>>>> .. other args ..
>>>>
>>>> (Windows or any platform)
>>>>
>>>> You can call the server java code directly: all one line:
>>>>
>>>>
>>>> java -Xmx1200M -jar fuseki-server.jar --set
>>>> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
>>>> args ..
>>>>
>>>> you'll need to put the full path name of fuseki-server.jar
>>>>
>>>> Sorry - I don't have your setup to test this fully. I did make sure that
>>>> the reworked query does lead to an execution plan that is different and
>>>> should yield some information about the situation.
>>>>
>>>> Andy
>>>>
>>>> On 22/12/15 09:50, Andy Seaborne wrote:
>>>>> On 22/12/15 07:06, Mark Wharton wrote:
>>>>>> Ah, wheels within wheels.
>>>>>>
>>>>>> The formulation with the filter in it is fine, except that if you want
>>>>>> to search for more than one word or you match in label and comment
>>>>>> then
>>>>>> the UNION formulation returns you duplicate rows. This isn't a
>>>>>> problem
>>>>>> with the Lucene search which is why (I now remember) I used it in the
>>>>>> first place.
>>>>>>
>>>>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>>>>> release at 2.3.0. Is there a way to find out?
>>>>>
>>>>> 3.0.0
>>>>>
>>>>> Many of the java commands support --version and the fuseki- server jar
>>>>> is an all-in-one jar:
>>>>>
>>>>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>>>>
>>>>>> What's the status on the JENA-999 and JENA-1093 issues? I see there's
>>>>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>>>>> comment seems encouraging.
>>>>>>
>>>>>> I don't want to adopt a single version as I'll be stuck forever
>>>>>> patching
>>>>>> back and forward and it will break eventually.
>>>>>>
>>>>>> Many thanks for your continued help.
>>>>>
>>>>> JENA-999 may sort of help but I'm not that positive because each ?ent
>>>>> from the first part will be different going into the second part. It
>>>>> looks to me as if it is the overhead of going out to Lucene. (This is
>>>>> Lucene right? not Solr?)
>>>>>
>>>>> The ideal is some super compilation of the text:query and spatial query
>>>>> into one big Lucene query.
>>>>>
>>>>> What would also be good, which is stop the general optimizer (this is
>>>>> nothing to do with TDB) using an index join. Except that is the better
>>>>> choice for the rdf:type. This is what the addition {} were trying for
>>>>> except the optimizer outsmarted
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> ?ent spatial:nearby( ...) .
>>>>> (?ent ?score) text:query (...) .
>>>>> ?ent rdf:type iotic:Entity .
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Mark - can you ask the query from Java? If so,
>>>>>
>>>>> Add "Optimize.noOptimizer(); " before executing the query. I can't
>>>>> see
>>>>> a way to do that from setting the environment for Fuseki.
>>>>>
>>>>> Or (the effect on time of this is version specific and whether it does
>>>>> anything useful is a big "maybe") you could try this:
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>>>>> { OPTIONAL { ?ent text:query "DEF" } }
>>>>> }
>>>>>
>>>>> Andy
>>>>>
>>>>
>>>>
>>
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi Andy.
More experiments this morning. I originally only send you a small part
of a larger query just to expose the problem in its simplest form. And
your switches work well in that case (i.e. first formulation below
*with* the comments.)
But... There's a problem when using the switches in that the rest of the
query wants to get the rdfs:label and various other properties. This
destroys the performance gains.
I've tried "yours" and "mine" with and without the switches and then the
separate parts on their own to see how that goes.
1) "yours"
==========
This formulation (with the switches and comments in place) - 384 ms
SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
WHERE {
{ ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.8018078804016'km') }
{ (?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
{?ent rdf:type iotic:Entity} }
# OPTIONAL {
# ?ent rdfs:label ?entLabel .
# FILTER langMatches( lang(?entLabel), 'en' ) .
# }
#
# OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
# ?ent iotic:Advertises ?point .
# ?point rdf:type iotic:Point .
# ?point iotic:PointType ?pointType .
#
# OPTIONAL {
# ?point rdfs:label ?pointLabel .
# FILTER langMatches( lang(?pointLabel), 'en' ) .
# }
}
Uncomment the lines and the performance drops to - 7.165 ms
2) "mine"
=========
The below formulation with the switches in place 11.221 secs
The below without the switches. 5.371 secs
SELECT ?score ?ent ?entLabel ?lat ?long ?point ?pointType ?pointLabel
WHERE {
?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.8018078804016'km') .
(?ent ?score) text:query ('environment' 'lang:en') .FILTER EXISTS
{?ent rdf:type iotic:Entity} .
OPTIONAL {
?ent rdfs:label ?entLabel .
FILTER langMatches( lang(?entLabel), 'en' ) .
}
OPTIONAL {?ent geo:lat ?lat . ?ent geo:long ?long}
?ent iotic:Advertises ?point .
?point rdf:type iotic:Point .
?point iotic:PointType ?pointType .
OPTIONAL {
?point rdfs:label ?pointLabel .
FILTER langMatches( lang(?pointLabel), 'en' ) .
}
}
3) Separately
==============
Completely on their own:
========================
i.e. just the ?ent spatial:nearby line
the spatial query on its own takes 50 ms
i.e just the text:query line
and the text on its own takes 258 ms
With the OPTIONAL {} and other properties
=========================================
Spatial and other properties 135 ms
Text and other properties 854 ms
Again, repeated thanks for you help.
Mark
Technology Lead, Iotic Labs
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 22/12/15 17:22, Andy Seaborne wrote:
> Mark,
>
> Thanks for the experiment results.
>
> On 22/12/15 15:47, Mark Wharton wrote:
>> Query below run without Andy's switches.
>> INFO [5] 200 OK (4.985 s)
>>
>> Query below run with Andy's switches.
>> INFO [1] 200 OK (840 ms)
>>
>> Them's some magic switches. Thanks, Andy.
>>
>> Do they have any impact (negative or positive) on any other SPARQL
>> operations? I'm only curious as you've solved our main problem in that
>> our "search" query was very slow. There's nowhere else that uses the
>> text and spatial indexes for retrieval.
>
> This depends on any internal change in the latest release (Jena 3.0.1,
> Fuseki 2.3.1). Prior to that it will not make the same difference.
> Specially, unoptimized joins are now hash joins.
>
> But that is not a good choice for the "?ent rdf:type iotic:Entity"
> triple pattern. The system can't distinguish different cases involving
> external indexes as it knows not very much about the index details.
>
> Adding
>
> FILTER EXISTS { ?ext rdf:type iotic:Entity }
>
> might work because the triple pattern is really a check, not a match
> setting a variable.
>
> A plain "?ent rdf:type iotic:Entity" will retrieve all things of that
> class regardless of spatial and text query when those optimization are off.
>
> Andy
>
>>
>> Many thanks for this help so close to the holiday season. Happy
>> holidays to you all at Jena - keep up the good work.
>>
>> Mark
>>
>>
>> Technology Lead, Iotic Labs
>> +44 7973 674404
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 22/12/15 11:49, Andy Seaborne wrote:
>>> Mark - here is another way.
>>>
>>> This query:
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { ?ent spatial:nearby ( .... ) }
>>> { ?ent text:query ( ..... ) }
>>> # No ?ent rdf:type iotic:Entity .
>>> # This focuses the query on the presenting issue.
>>> }
>>>
>>> and then run Fuseki with the following flags:
>>>
>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>>>
>>> for however you are running the server.
>>>
>>> You need both --set
>>>
>>> The service script will not do this very easily - if environment
>>> variable FUSEKI_ARGS is set it might do. Untested.
>>>
>>> It is easier to run the server standalone:
>>>
>>> (Linux, Mac)
>>>
>>> The "fuseki-server" script should pass these in:
>>>
>>> fuseki-server \
>>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false \
>>> .. other args ..
>>>
>>> (Windows or any platform)
>>>
>>> You can call the server java code directly: all one line:
>>>
>>>
>>> java -Xmx1200M -jar fuseki-server.jar --set
>>> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
>>> args ..
>>>
>>> you'll need to put the full path name of fuseki-server.jar
>>>
>>> Sorry - I don't have your setup to test this fully. I did make sure that
>>> the reworked query does lead to an execution plan that is different and
>>> should yield some information about the situation.
>>>
>>> Andy
>>>
>>> On 22/12/15 09:50, Andy Seaborne wrote:
>>>> On 22/12/15 07:06, Mark Wharton wrote:
>>>>> Ah, wheels within wheels.
>>>>>
>>>>> The formulation with the filter in it is fine, except that if you want
>>>>> to search for more than one word or you match in label and comment
>>>>> then
>>>>> the UNION formulation returns you duplicate rows. This isn't a
>>>>> problem
>>>>> with the Lucene search which is why (I now remember) I used it in the
>>>>> first place.
>>>>>
>>>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>>>> release at 2.3.0. Is there a way to find out?
>>>>
>>>> 3.0.0
>>>>
>>>> Many of the java commands support --version and the fuseki- server jar
>>>> is an all-in-one jar:
>>>>
>>>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>>>
>>>>> What's the status on the JENA-999 and JENA-1093 issues? I see there's
>>>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>>>> comment seems encouraging.
>>>>>
>>>>> I don't want to adopt a single version as I'll be stuck forever
>>>>> patching
>>>>> back and forward and it will break eventually.
>>>>>
>>>>> Many thanks for your continued help.
>>>>
>>>> JENA-999 may sort of help but I'm not that positive because each ?ent
>>>> from the first part will be different going into the second part. It
>>>> looks to me as if it is the overhead of going out to Lucene. (This is
>>>> Lucene right? not Solr?)
>>>>
>>>> The ideal is some super compilation of the text:query and spatial query
>>>> into one big Lucene query.
>>>>
>>>> What would also be good, which is stop the general optimizer (this is
>>>> nothing to do with TDB) using an index join. Except that is the better
>>>> choice for the rdf:type. This is what the addition {} were trying for
>>>> except the optimizer outsmarted
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> ?ent spatial:nearby( ...) .
>>>> (?ent ?score) text:query (...) .
>>>> ?ent rdf:type iotic:Entity .
>>>> }
>>>>
>>>>
>>>>
>>>> Mark - can you ask the query from Java? If so,
>>>>
>>>> Add "Optimize.noOptimizer(); " before executing the query. I can't
>>>> see
>>>> a way to do that from setting the environment for Fuseki.
>>>>
>>>> Or (the effect on time of this is version specific and whether it does
>>>> anything useful is a big "maybe") you could try this:
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>>>> { OPTIONAL { ?ent text:query "DEF" } }
>>>> }
>>>>
>>>> Andy
>>>>
>>>
>>>
>
Re: Query with spatial *and* text searches.
Posted by Andy Seaborne <an...@apache.org>.
Mark,
Thanks for the experiment results.
On 22/12/15 15:47, Mark Wharton wrote:
> Query below run without Andy's switches.
> INFO [5] 200 OK (4.985 s)
>
> Query below run with Andy's switches.
> INFO [1] 200 OK (840 ms)
>
> Them's some magic switches. Thanks, Andy.
>
> Do they have any impact (negative or positive) on any other SPARQL
> operations? I'm only curious as you've solved our main problem in that
> our "search" query was very slow. There's nowhere else that uses the
> text and spatial indexes for retrieval.
This depends on any internal change in the latest release (Jena 3.0.1,
Fuseki 2.3.1). Prior to that it will not make the same difference.
Specially, unoptimized joins are now hash joins.
But that is not a good choice for the "?ent rdf:type iotic:Entity"
triple pattern. The system can't distinguish different cases involving
external indexes as it knows not very much about the index details.
Adding
FILTER EXISTS { ?ext rdf:type iotic:Entity }
might work because the triple pattern is really a check, not a match
setting a variable.
A plain "?ent rdf:type iotic:Entity" will retrieve all things of that
class regardless of spatial and text query when those optimization are off.
Andy
>
> Many thanks for this help so close to the holiday season. Happy
> holidays to you all at Jena - keep up the good work.
>
> Mark
>
>
> Technology Lead, Iotic Labs
> +44 7973 674404
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 22/12/15 11:49, Andy Seaborne wrote:
>> Mark - here is another way.
>>
>> This query:
>>
>> SELECT ?score ?ent
>> WHERE {
>> { ?ent spatial:nearby ( .... ) }
>> { ?ent text:query ( ..... ) }
>> # No ?ent rdf:type iotic:Entity .
>> # This focuses the query on the presenting issue.
>> }
>>
>> and then run Fuseki with the following flags:
>>
>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>>
>> for however you are running the server.
>>
>> You need both --set
>>
>> The service script will not do this very easily - if environment
>> variable FUSEKI_ARGS is set it might do. Untested.
>>
>> It is easier to run the server standalone:
>>
>> (Linux, Mac)
>>
>> The "fuseki-server" script should pass these in:
>>
>> fuseki-server \
>> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false \
>> .. other args ..
>>
>> (Windows or any platform)
>>
>> You can call the server java code directly: all one line:
>>
>>
>> java -Xmx1200M -jar fuseki-server.jar --set
>> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
>> args ..
>>
>> you'll need to put the full path name of fuseki-server.jar
>>
>> Sorry - I don't have your setup to test this fully. I did make sure that
>> the reworked query does lead to an execution plan that is different and
>> should yield some information about the situation.
>>
>> Andy
>>
>> On 22/12/15 09:50, Andy Seaborne wrote:
>>> On 22/12/15 07:06, Mark Wharton wrote:
>>>> Ah, wheels within wheels.
>>>>
>>>> The formulation with the filter in it is fine, except that if you want
>>>> to search for more than one word or you match in label and comment then
>>>> the UNION formulation returns you duplicate rows. This isn't a problem
>>>> with the Lucene search which is why (I now remember) I used it in the
>>>> first place.
>>>>
>>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>>> release at 2.3.0. Is there a way to find out?
>>>
>>> 3.0.0
>>>
>>> Many of the java commands support --version and the fuseki- server jar
>>> is an all-in-one jar:
>>>
>>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>>
>>>> What's the status on the JENA-999 and JENA-1093 issues? I see there's
>>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>>> comment seems encouraging.
>>>>
>>>> I don't want to adopt a single version as I'll be stuck forever patching
>>>> back and forward and it will break eventually.
>>>>
>>>> Many thanks for your continued help.
>>>
>>> JENA-999 may sort of help but I'm not that positive because each ?ent
>>> from the first part will be different going into the second part. It
>>> looks to me as if it is the overhead of going out to Lucene. (This is
>>> Lucene right? not Solr?)
>>>
>>> The ideal is some super compilation of the text:query and spatial query
>>> into one big Lucene query.
>>>
>>> What would also be good, which is stop the general optimizer (this is
>>> nothing to do with TDB) using an index join. Except that is the better
>>> choice for the rdf:type. This is what the addition {} were trying for
>>> except the optimizer outsmarted
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> ?ent spatial:nearby( ...) .
>>> (?ent ?score) text:query (...) .
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>>
>>>
>>> Mark - can you ask the query from Java? If so,
>>>
>>> Add "Optimize.noOptimizer(); " before executing the query. I can't see
>>> a way to do that from setting the environment for Fuseki.
>>>
>>> Or (the effect on time of this is version specific and whether it does
>>> anything useful is a big "maybe") you could try this:
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>>> { OPTIONAL { ?ent text:query "DEF" } }
>>> }
>>>
>>> Andy
>>>
>>
>>
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Query below run without Andy's switches.
INFO [5] 200 OK (4.985 s)
Query below run with Andy's switches.
INFO [1] 200 OK (840 ms)
Them's some magic switches. Thanks, Andy.
Do they have any impact (negative or positive) on any other SPARQL
operations? I'm only curious as you've solved our main problem in that
our "search" query was very slow. There's nowhere else that uses the
text and spatial indexes for retrieval.
Many thanks for this help so close to the holiday season. Happy
holidays to you all at Jena - keep up the good work.
Mark
Technology Lead, Iotic Labs
+44 7973 674404
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 22/12/15 11:49, Andy Seaborne wrote:
> Mark - here is another way.
>
> This query:
>
> SELECT ?score ?ent
> WHERE {
> { ?ent spatial:nearby ( .... ) }
> { ?ent text:query ( ..... ) }
> # No ?ent rdf:type iotic:Entity .
> # This focuses the query on the presenting issue.
> }
>
> and then run Fuseki with the following flags:
>
> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
>
> for however you are running the server.
>
> You need both --set
>
> The service script will not do this very easily - if environment
> variable FUSEKI_ARGS is set it might do. Untested.
>
> It is easier to run the server standalone:
>
> (Linux, Mac)
>
> The "fuseki-server" script should pass these in:
>
> fuseki-server \
> --set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false \
> .. other args ..
>
> (Windows or any platform)
>
> You can call the server java code directly: all one line:
>
>
> java -Xmx1200M -jar fuseki-server.jar --set
> arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other
> args ..
>
> you'll need to put the full path name of fuseki-server.jar
>
> Sorry - I don't have your setup to test this fully. I did make sure that
> the reworked query does lead to an execution plan that is different and
> should yield some information about the situation.
>
> Andy
>
> On 22/12/15 09:50, Andy Seaborne wrote:
>> On 22/12/15 07:06, Mark Wharton wrote:
>>> Ah, wheels within wheels.
>>>
>>> The formulation with the filter in it is fine, except that if you want
>>> to search for more than one word or you match in label and comment then
>>> the UNION formulation returns you duplicate rows. This isn't a problem
>>> with the Lucene search which is why (I now remember) I used it in the
>>> first place.
>>>
>>> I'm not sure what version of jena I'm using - I just use the fuseki
>>> release at 2.3.0. Is there a way to find out?
>>
>> 3.0.0
>>
>> Many of the java commands support --version and the fuseki- server jar
>> is an all-in-one jar:
>>
>> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>>
>>> What's the status on the JENA-999 and JENA-1093 issues? I see there's
>>> been some activity on 999 in the last few days. Andy Seaborne's last
>>> comment seems encouraging.
>>>
>>> I don't want to adopt a single version as I'll be stuck forever patching
>>> back and forward and it will break eventually.
>>>
>>> Many thanks for your continued help.
>>
>> JENA-999 may sort of help but I'm not that positive because each ?ent
>> from the first part will be different going into the second part. It
>> looks to me as if it is the overhead of going out to Lucene. (This is
>> Lucene right? not Solr?)
>>
>> The ideal is some super compilation of the text:query and spatial query
>> into one big Lucene query.
>>
>> What would also be good, which is stop the general optimizer (this is
>> nothing to do with TDB) using an index join. Except that is the better
>> choice for the rdf:type. This is what the addition {} were trying for
>> except the optimizer outsmarted
>>
>> SELECT ?score ?ent
>> WHERE {
>> ?ent spatial:nearby( ...) .
>> (?ent ?score) text:query (...) .
>> ?ent rdf:type iotic:Entity .
>> }
>>
>>
>>
>> Mark - can you ask the query from Java? If so,
>>
>> Add "Optimize.noOptimizer(); " before executing the query. I can't see
>> a way to do that from setting the environment for Fuseki.
>>
>> Or (the effect on time of this is version specific and whether it does
>> anything useful is a big "maybe") you could try this:
>>
>> SELECT ?score ?ent
>> WHERE {
>> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
>> { OPTIONAL { ?ent text:query "DEF" } }
>> }
>>
>> Andy
>>
>
>
Re: Query with spatial *and* text searches.
Posted by Andy Seaborne <an...@apache.org>.
Mark - here is another way.
This query:
SELECT ?score ?ent
WHERE {
{ ?ent spatial:nearby ( .... ) }
{ ?ent text:query ( ..... ) }
# No ?ent rdf:type iotic:Entity .
# This focuses the query on the presenting issue.
}
and then run Fuseki with the following flags:
--set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false
for however you are running the server.
You need both --set
The service script will not do this very easily - if environment
variable FUSEKI_ARGS is set it might do. Untested.
It is easier to run the server standalone:
(Linux, Mac)
The "fuseki-server" script should pass these in:
fuseki-server \
--set arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false \
.. other args ..
(Windows or any platform)
You can call the server java code directly: all one line:
java -Xmx1200M -jar fuseki-server.jar --set
arq:optIndexJoinStrategy=false --set arq:optMergeBGPs=false .. other args ..
you'll need to put the full path name of fuseki-server.jar
Sorry - I don't have your setup to test this fully. I did make sure that
the reworked query does lead to an execution plan that is different and
should yield some information about the situation.
Andy
On 22/12/15 09:50, Andy Seaborne wrote:
> On 22/12/15 07:06, Mark Wharton wrote:
>> Ah, wheels within wheels.
>>
>> The formulation with the filter in it is fine, except that if you want
>> to search for more than one word or you match in label and comment then
>> the UNION formulation returns you duplicate rows. This isn't a problem
>> with the Lucene search which is why (I now remember) I used it in the
>> first place.
>>
>> I'm not sure what version of jena I'm using - I just use the fuseki
>> release at 2.3.0. Is there a way to find out?
>
> 3.0.0
>
> Many of the java commands support --version and the fuseki- server jar
> is an all-in-one jar:
>
> java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
>
>> What's the status on the JENA-999 and JENA-1093 issues? I see there's
>> been some activity on 999 in the last few days. Andy Seaborne's last
>> comment seems encouraging.
>>
>> I don't want to adopt a single version as I'll be stuck forever patching
>> back and forward and it will break eventually.
>>
>> Many thanks for your continued help.
>
> JENA-999 may sort of help but I'm not that positive because each ?ent
> from the first part will be different going into the second part. It
> looks to me as if it is the overhead of going out to Lucene. (This is
> Lucene right? not Solr?)
>
> The ideal is some super compilation of the text:query and spatial query
> into one big Lucene query.
>
> What would also be good, which is stop the general optimizer (this is
> nothing to do with TDB) using an index join. Except that is the better
> choice for the rdf:type. This is what the addition {} were trying for
> except the optimizer outsmarted
>
> SELECT ?score ?ent
> WHERE {
> ?ent spatial:nearby( ...) .
> (?ent ?score) text:query (...) .
> ?ent rdf:type iotic:Entity .
> }
>
>
>
> Mark - can you ask the query from Java? If so,
>
> Add "Optimize.noOptimizer(); " before executing the query. I can't see
> a way to do that from setting the environment for Fuseki.
>
> Or (the effect on time of this is version specific and whether it does
> anything useful is a big "maybe") you could try this:
>
> SELECT ?score ?ent
> WHERE {
> { OPTIONAL { ?ent spatial:nearby "ABC" . }}
> { OPTIONAL { ?ent text:query "DEF" } }
> }
>
> Andy
>
Re: Query with spatial *and* text searches.
Posted by Andy Seaborne <an...@apache.org>.
On 22/12/15 07:06, Mark Wharton wrote:
> Ah, wheels within wheels.
>
> The formulation with the filter in it is fine, except that if you want
> to search for more than one word or you match in label and comment then
> the UNION formulation returns you duplicate rows. This isn't a problem
> with the Lucene search which is why (I now remember) I used it in the
> first place.
>
> I'm not sure what version of jena I'm using - I just use the fuseki
> release at 2.3.0. Is there a way to find out?
3.0.0
Many of the java commands support --version and the fuseki- server jar
is an all-in-one jar:
java -cp <YourInstall>/fuseki-server.jar arq.sparql --version
> What's the status on the JENA-999 and JENA-1093 issues? I see there's
> been some activity on 999 in the last few days. Andy Seaborne's last
> comment seems encouraging.
>
> I don't want to adopt a single version as I'll be stuck forever patching
> back and forward and it will break eventually.
>
> Many thanks for your continued help.
JENA-999 may sort of help but I'm not that positive because each ?ent
from the first part will be different going into the second part. It
looks to me as if it is the overhead of going out to Lucene. (This is
Lucene right? not Solr?)
The ideal is some super compilation of the text:query and spatial query
into one big Lucene query.
What would also be good, which is stop the general optimizer (this is
nothing to do with TDB) using an index join. Except that is the better
choice for the rdf:type. This is what the addition {} were trying for
except the optimizer outsmarted
SELECT ?score ?ent
WHERE {
?ent spatial:nearby( ...) .
(?ent ?score) text:query (...) .
?ent rdf:type iotic:Entity .
}
Mark - can you ask the query from Java? If so,
Add "Optimize.noOptimizer(); " before executing the query. I can't see
a way to do that from setting the environment for Fuseki.
Or (the effect on time of this is version specific and whether it does
anything useful is a big "maybe") you could try this:
SELECT ?score ?ent
WHERE {
{ OPTIONAL { ?ent spatial:nearby "ABC" . }}
{ OPTIONAL { ?ent text:query "DEF" } }
}
Andy
>
> Mark
>
> Technology Lead, Iotic Labs
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 21/12/15 16:04, Mark Wharton wrote:
>> Hi Osma.
>>
>> Please don't apologise. It's nice to have someone to discuss this with
>> who is so far up the curve.
>>
>> I was just going to rewrite exactly how you suggest (giving in?) but you
>> beat me to it.
>>
>> SELECT ?score ?ent
>> WHERE {
>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.01807880401611 'km') .
>> {
>> { ?ent rdfs:label ?label .
>> FILTER (regex(?label, "environment", "i") &&
>> langmatches(lang(?label), "en"))
>> }
>> UNION
>> { ?ent rdfs:comment ?comment .
>> FILTER (regex(?comment, "environment", "i") &&
>> langmatches(lang(?comment), "en")) }
>> }
>> ?ent rdf:type iotic:Entity .
>> }
>>
>> gets the same results (sans ?score, but who cares?) and completes in
>> 61ms! No contest.
>>
>> Thanks for your help
>>
>> Mark
>>
>> Technology Lead, Iotic Labs
>> +44 7973 674404
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 21/12/15 14:30, Osma Suominen wrote:
>>> Hi Mark!
>>>
>>> Thanks for trying my queries. I'm sorry (but not surprised!) to hear
>>> that they weren't any better than your original query.
>>>
>>> I think JENA-999 is really the key here - until it is implemented, I
>>> don't know any way of speeding this up. My (very fuzzy) understanding of
>>> ARQ is that it always performs joins with iterators, so there is no way
>>> to first retrieve both result sets (one from spatial, one from
>>> jena-text) and only then join them, unless you do the cross product and
>>> only limit with FILTER, which is even slower (options 2 and 3).
>>>
>>> By the way you didn't say which version of Jena you're using. Jena 3.0.1
>>> actually does include the initial JENA-999 patch which, despite its
>>> problems (related to retrieving score and especially the original
>>> literal - see JENA-1093), might actually work in your case. It was
>>> reverted only after the 3.0.1 release.
>>>
>>> Other than testing with 3.0.1, I'd suggest skipping jena-text for now
>>> and changing the text query part to a FILTER expression instead, with a
>>> REGEX or CONTAINS. Applying that kind of filter to 450 items shouldn't
>>> add much to the execution time of the spatial query.
>>>
>>> -Osma
>>>
>>> [1] https://issues.apache.org/jira/browse/JENA-999
>>> [2] https://issues.apache.org/jira/browse/JENA-1093
>>>
>>> On 21/12/15 13:02, Mark Wharton wrote:
>>>> Hi Osma.
>>>>
>>>> Thanks for your help. It was exactly the kind of help that I wanted.
>>>>
>>>> Haha, but life is rarely that simple...
>>>>
>>>> I ran all your versions and my original a couple of times each to see if
>>>> there was any difference in performance.
>>>>
>>>> query time1 time2
>>>> Orig 4.615 4.632
>>>> 1 4.648 4.608
>>>> 2 6.385 6.401
>>>> 3 6.353 6.442
>>>>
>>>> I think that proves that my "naive" version is working how I thought,
>>>> (and the same as your version 1). i.e. getting all the results from the
>>>> spatial predicate and then individually checking them against the text
>>>> predicate. (Which is why it's slower if you swap the predicates over,
>>>> as there are more matches on the text).
>>>>
>>>> This kind of query can't be that unusual, surely? Things with "foo" in
>>>> their text within 50Km of point "bar"?
>>>>
>>>> Mark
>>>>
>>>>
>>>> Technology Lead, Iotic Labs
>>>> +44 7973 674404
>>>> mark.wharton@iotic-labs.com
>>>> https://www.iotic-labs.com
>>>>
>>>> On 21/12/15 07:53, Osma Suominen wrote:
>>>>> Hi Mark!
>>>>>
>>>>> I'm not sure that the jena-external-index approach would help. It might
>>>>> or might not, depending on how it's implemented. AFAIK it's just an idea
>>>>> right now, I haven't seen any code.
>>>>>
>>>>> In any case I think the problem with jena-text and probably jena-spatial
>>>>> too (not very familiar with it) is that they are pretty fast when you
>>>>> can get by doing just a single query with an unbound subject. But when
>>>>> you instead have a fixed subject, or more likely a list of possible
>>>>> subjects, the performance will be very bad because every subject will be
>>>>> queried separately from the Lucene index. This is probably what happens
>>>>> with your query - the results of one query will be fed to the other.
>>>>> JENA-999 tried to address some of this by introducing a cache into
>>>>> jena-text, but the patch that was committed had other problems (it
>>>>> returned the wrong results in some cases) so it was reverted recently
>>>>> and an improved version hasn't yet been developed.
>>>>>
>>>>> In any case, you could try these variants of the original query:
>>>>> (I'm really on thin ground here, but I spent a couple of days last week
>>>>> trying to optimize a somewhat similar query involving jena-text and
>>>>> various other conditions, trying to find a solution with good
>>>>> performance)
>>>>>
>>>>> 1. Try to isolate the text and spatial queries with extra braces
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>> 70.01807880401611 'km') }
>>>>> { (?ent ?score) text:query ('environment' 'lang:en') }
>>>>> ?ent rdf:type iotic:Entity .
>>>>> }
>>>>>
>>>>> 2. Use a FILTER expression to delay matching the results
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>> 70.01807880401611 'km') .
>>>>> (?ent2 ?score) text:query ('environment' 'lang:en') .
>>>>> FILTER (?ent = ?ent2)
>>>>> ?ent rdf:type iotic:Entity .
>>>>> }
>>>>>
>>>>> 3. Combination of above ideas
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>> 70.01807880401611 'km') }
>>>>> { (?ent2 ?score) text:query ('environment' 'lang:en') }
>>>>> FILTER (?ent = ?ent2)
>>>>> ?ent rdf:type iotic:Entity .
>>>>> }
>>>>>
>>>>>
>>>>> -Osma
>>>>>
>>>>>
>>>>> On 21/12/15 00:28, Mark Wharton wrote:
>>>>>> Hi Marco.
>>>>>>
>>>>>> Yes, that's it. The indexes work well in isolation, but don't combine
>>>>>> well. Smooshing them into a single index would be a great idea,
>>>>>> especially if the query could resolve both text and spatial predicates
>>>>>> with one matching scan of the index.
>>>>>>
>>>>>> Perhaps Stephen could be persuaded to pick up the pace on this one?
>>>>>>
>>>>>> Thanks Mark
>>>>>>
>>>>>> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann
>>>>>> <ma...@gmail.com> wrote:
>>>>>>> yes correct Mark I am only referring to the extra payload here for
>>>>>>> invoking the spatial filter in the SPARQL query.
>>>>>>>
>>>>>>> now that you mention a particular issue with the combined use of both
>>>>>>> jena-text and jena-spatial (something I am not aware of ) this might
>>>>>>> be related to duplicated code in the two projects. back in May Stephen
>>>>>>> Allen wrote on the dev-list that he is about to address some of this
>>>>>>> possibly in a new jena-external-index project.
>>>>>>>
>>>>>>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>>>>>>> <ma...@iotic-labs.com> wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Thanks for this. I've read the chapter in the book and now I'm not
>>>>>>> sure
>>>>>>>> if I misunderstand your reply or you've only addressed half of the
>>>>>>> problem.
>>>>>>>>
>>>>>>>> I'm not worried about the performance of the spatial search in
>>>>>>> isolation
>>>>>>>> - that's 97ms which is fine. The text search on its own takes a bit
>>>>>>>> longer but that's acceptable, too.
>>>>>>>>
>>>>>>>> It's when I put the spatial and text *together* that query time
>>>>>>> increase
>>>>>>>> by 10-30 times. That's the bit I don't understand and would like
>>>>>>> some
>>>>>>>> help with.
>>>>>>>>
>>>>>>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>>>>>>> than retrieving one set and looping through to retrieve the matches
>>>>>>>> individually on the other. (Which is my guess as to how it works).
>>>>>>>>
>>>>>>>> Thanks for your help so far.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>> Technology Lead, Iotic Labs
>>>>>>>> mark.wharton@iotic-labs.com
>>>>>>>> https://www.iotic-labs.com
>>>>>>>>
>>>>>>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>>>>>>> it's a common spatial access method latency in paticular for small
>>>>>>>>> data sets. you can try a mbr range query instead.
>>>>>>>>>
>>>>>>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>>>>>>> by
>>>>>>>>> John Hebeler et. al.. 2009
>>>>>>>>>
>>>>>>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>>>>>>> <ma...@iotic-labs.com> wrote:
>>>>>>>>>> Hi Jena users.
>>>>>>>>>>
>>>>>>>>>> I'm having performance problems with a query that uses text and
>>>>>>> location
>>>>>>>>>> search
>>>>>>>>>>
>>>>>>>>>> The query is roughly this:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> SELECT ?score ?ent
>>>>>>>>>> WHERE {
>>>>>>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>>>>>>> 70.01807880401611 'km') .
>>>>>>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There are about 450 entities in that radius
>>>>>>>>>> There are about 2200 entities with environment in their
>>>>>>> rdfs:comment
>>>>>>>>>>
>>>>>>>>>> The query takes 5 seconds.
>>>>>>>>>>
>>>>>>>>>> I've tried this:
>>>>>>>>>> Commenting out the text predicate the query takes 97 ms
>>>>>>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>>>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> My question is this... It looks like the query is separately
>>>>>>> getting
>>>>>>>>>> the results of the first two predicates and merging (somehow) to
>>>>>>> find
>>>>>>>>>> the intersection. Is there a formulation which will intersect the
>>>>>>> two
>>>>>>>>>> sets faster?
>>>>>>>>>>
>>>>>>>>>> Many TIAs,
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>> --
>>>>>>>>>> Technology Lead, Iotic Labs
>>>>>>>>>> mark.wharton@iotic-labs.com
>>>>>>>>>> https://www.iotic-labs.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Marco Neumann
>>>>>>> KONA
>>>>>>
>>>>>
>>>>>
>>>
>>>
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Ah, wheels within wheels.
The formulation with the filter in it is fine, except that if you want
to search for more than one word or you match in label and comment then
the UNION formulation returns you duplicate rows. This isn't a problem
with the Lucene search which is why (I now remember) I used it in the
first place.
I'm not sure what version of jena I'm using - I just use the fuseki
release at 2.3.0. Is there a way to find out?
What's the status on the JENA-999 and JENA-1093 issues? I see there's
been some activity on 999 in the last few days. Andy Seaborne's last
comment seems encouraging.
I don't want to adopt a single version as I'll be stuck forever patching
back and forward and it will break eventually.
Many thanks for your continued help.
Mark
Technology Lead, Iotic Labs
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 21/12/15 16:04, Mark Wharton wrote:
> Hi Osma.
>
> Please don't apologise. It's nice to have someone to discuss this with
> who is so far up the curve.
>
> I was just going to rewrite exactly how you suggest (giving in?) but you
> beat me to it.
>
> SELECT ?score ?ent
> WHERE {
> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.01807880401611 'km') .
> {
> { ?ent rdfs:label ?label .
> FILTER (regex(?label, "environment", "i") &&
> langmatches(lang(?label), "en"))
> }
> UNION
> { ?ent rdfs:comment ?comment .
> FILTER (regex(?comment, "environment", "i") &&
> langmatches(lang(?comment), "en")) }
> }
> ?ent rdf:type iotic:Entity .
> }
>
> gets the same results (sans ?score, but who cares?) and completes in
> 61ms! No contest.
>
> Thanks for your help
>
> Mark
>
> Technology Lead, Iotic Labs
> +44 7973 674404
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 21/12/15 14:30, Osma Suominen wrote:
>> Hi Mark!
>>
>> Thanks for trying my queries. I'm sorry (but not surprised!) to hear
>> that they weren't any better than your original query.
>>
>> I think JENA-999 is really the key here - until it is implemented, I
>> don't know any way of speeding this up. My (very fuzzy) understanding of
>> ARQ is that it always performs joins with iterators, so there is no way
>> to first retrieve both result sets (one from spatial, one from
>> jena-text) and only then join them, unless you do the cross product and
>> only limit with FILTER, which is even slower (options 2 and 3).
>>
>> By the way you didn't say which version of Jena you're using. Jena 3.0.1
>> actually does include the initial JENA-999 patch which, despite its
>> problems (related to retrieving score and especially the original
>> literal - see JENA-1093), might actually work in your case. It was
>> reverted only after the 3.0.1 release.
>>
>> Other than testing with 3.0.1, I'd suggest skipping jena-text for now
>> and changing the text query part to a FILTER expression instead, with a
>> REGEX or CONTAINS. Applying that kind of filter to 450 items shouldn't
>> add much to the execution time of the spatial query.
>>
>> -Osma
>>
>> [1] https://issues.apache.org/jira/browse/JENA-999
>> [2] https://issues.apache.org/jira/browse/JENA-1093
>>
>> On 21/12/15 13:02, Mark Wharton wrote:
>>> Hi Osma.
>>>
>>> Thanks for your help. It was exactly the kind of help that I wanted.
>>>
>>> Haha, but life is rarely that simple...
>>>
>>> I ran all your versions and my original a couple of times each to see if
>>> there was any difference in performance.
>>>
>>> query time1 time2
>>> Orig 4.615 4.632
>>> 1 4.648 4.608
>>> 2 6.385 6.401
>>> 3 6.353 6.442
>>>
>>> I think that proves that my "naive" version is working how I thought,
>>> (and the same as your version 1). i.e. getting all the results from the
>>> spatial predicate and then individually checking them against the text
>>> predicate. (Which is why it's slower if you swap the predicates over,
>>> as there are more matches on the text).
>>>
>>> This kind of query can't be that unusual, surely? Things with "foo" in
>>> their text within 50Km of point "bar"?
>>>
>>> Mark
>>>
>>>
>>> Technology Lead, Iotic Labs
>>> +44 7973 674404
>>> mark.wharton@iotic-labs.com
>>> https://www.iotic-labs.com
>>>
>>> On 21/12/15 07:53, Osma Suominen wrote:
>>>> Hi Mark!
>>>>
>>>> I'm not sure that the jena-external-index approach would help. It might
>>>> or might not, depending on how it's implemented. AFAIK it's just an idea
>>>> right now, I haven't seen any code.
>>>>
>>>> In any case I think the problem with jena-text and probably jena-spatial
>>>> too (not very familiar with it) is that they are pretty fast when you
>>>> can get by doing just a single query with an unbound subject. But when
>>>> you instead have a fixed subject, or more likely a list of possible
>>>> subjects, the performance will be very bad because every subject will be
>>>> queried separately from the Lucene index. This is probably what happens
>>>> with your query - the results of one query will be fed to the other.
>>>> JENA-999 tried to address some of this by introducing a cache into
>>>> jena-text, but the patch that was committed had other problems (it
>>>> returned the wrong results in some cases) so it was reverted recently
>>>> and an improved version hasn't yet been developed.
>>>>
>>>> In any case, you could try these variants of the original query:
>>>> (I'm really on thin ground here, but I spent a couple of days last week
>>>> trying to optimize a somewhat similar query involving jena-text and
>>>> various other conditions, trying to find a solution with good
>>>> performance)
>>>>
>>>> 1. Try to isolate the text and spatial queries with extra braces
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>> 70.01807880401611 'km') }
>>>> { (?ent ?score) text:query ('environment' 'lang:en') }
>>>> ?ent rdf:type iotic:Entity .
>>>> }
>>>>
>>>> 2. Use a FILTER expression to delay matching the results
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>> 70.01807880401611 'km') .
>>>> (?ent2 ?score) text:query ('environment' 'lang:en') .
>>>> FILTER (?ent = ?ent2)
>>>> ?ent rdf:type iotic:Entity .
>>>> }
>>>>
>>>> 3. Combination of above ideas
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>> 70.01807880401611 'km') }
>>>> { (?ent2 ?score) text:query ('environment' 'lang:en') }
>>>> FILTER (?ent = ?ent2)
>>>> ?ent rdf:type iotic:Entity .
>>>> }
>>>>
>>>>
>>>> -Osma
>>>>
>>>>
>>>> On 21/12/15 00:28, Mark Wharton wrote:
>>>>> Hi Marco.
>>>>>
>>>>> Yes, that's it. The indexes work well in isolation, but don't combine
>>>>> well. Smooshing them into a single index would be a great idea,
>>>>> especially if the query could resolve both text and spatial predicates
>>>>> with one matching scan of the index.
>>>>>
>>>>> Perhaps Stephen could be persuaded to pick up the pace on this one?
>>>>>
>>>>> Thanks Mark
>>>>>
>>>>> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann
>>>>> <ma...@gmail.com> wrote:
>>>>>> yes correct Mark I am only referring to the extra payload here for
>>>>>> invoking the spatial filter in the SPARQL query.
>>>>>>
>>>>>> now that you mention a particular issue with the combined use of both
>>>>>> jena-text and jena-spatial (something I am not aware of ) this might
>>>>>> be related to duplicated code in the two projects. back in May Stephen
>>>>>> Allen wrote on the dev-list that he is about to address some of this
>>>>>> possibly in a new jena-external-index project.
>>>>>>
>>>>>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>>>>>> <ma...@iotic-labs.com> wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Thanks for this. I've read the chapter in the book and now I'm not
>>>>>> sure
>>>>>>> if I misunderstand your reply or you've only addressed half of the
>>>>>> problem.
>>>>>>>
>>>>>>> I'm not worried about the performance of the spatial search in
>>>>>> isolation
>>>>>>> - that's 97ms which is fine. The text search on its own takes a bit
>>>>>>> longer but that's acceptable, too.
>>>>>>>
>>>>>>> It's when I put the spatial and text *together* that query time
>>>>>> increase
>>>>>>> by 10-30 times. That's the bit I don't understand and would like
>>>>>> some
>>>>>>> help with.
>>>>>>>
>>>>>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>>>>>> than retrieving one set and looping through to retrieve the matches
>>>>>>> individually on the other. (Which is my guess as to how it works).
>>>>>>>
>>>>>>> Thanks for your help so far.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> Technology Lead, Iotic Labs
>>>>>>> mark.wharton@iotic-labs.com
>>>>>>> https://www.iotic-labs.com
>>>>>>>
>>>>>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>>>>>> it's a common spatial access method latency in paticular for small
>>>>>>>> data sets. you can try a mbr range query instead.
>>>>>>>>
>>>>>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>>>>>> by
>>>>>>>> John Hebeler et. al.. 2009
>>>>>>>>
>>>>>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>>>>>> <ma...@iotic-labs.com> wrote:
>>>>>>>>> Hi Jena users.
>>>>>>>>>
>>>>>>>>> I'm having performance problems with a query that uses text and
>>>>>> location
>>>>>>>>> search
>>>>>>>>>
>>>>>>>>> The query is roughly this:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SELECT ?score ?ent
>>>>>>>>> WHERE {
>>>>>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>>>>>> 70.01807880401611 'km') .
>>>>>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There are about 450 entities in that radius
>>>>>>>>> There are about 2200 entities with environment in their
>>>>>> rdfs:comment
>>>>>>>>>
>>>>>>>>> The query takes 5 seconds.
>>>>>>>>>
>>>>>>>>> I've tried this:
>>>>>>>>> Commenting out the text predicate the query takes 97 ms
>>>>>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My question is this... It looks like the query is separately
>>>>>> getting
>>>>>>>>> the results of the first two predicates and merging (somehow) to
>>>>>> find
>>>>>>>>> the intersection. Is there a formulation which will intersect the
>>>>>> two
>>>>>>>>> sets faster?
>>>>>>>>>
>>>>>>>>> Many TIAs,
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>> --
>>>>>>>>> Technology Lead, Iotic Labs
>>>>>>>>> mark.wharton@iotic-labs.com
>>>>>>>>> https://www.iotic-labs.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Marco Neumann
>>>>>> KONA
>>>>>
>>>>
>>>>
>>
>>
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi Osma.
Please don't apologise. It's nice to have someone to discuss this with
who is so far up the curve.
I was just going to rewrite exactly how you suggest (giving in?) but you
beat me to it.
SELECT ?score ?ent
WHERE {
?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.01807880401611 'km') .
{
{ ?ent rdfs:label ?label .
FILTER (regex(?label, "environment", "i") &&
langmatches(lang(?label), "en"))
}
UNION
{ ?ent rdfs:comment ?comment .
FILTER (regex(?comment, "environment", "i") &&
langmatches(lang(?comment), "en")) }
}
?ent rdf:type iotic:Entity .
}
gets the same results (sans ?score, but who cares?) and completes in
61ms! No contest.
Thanks for your help
Mark
Technology Lead, Iotic Labs
+44 7973 674404
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 21/12/15 14:30, Osma Suominen wrote:
> Hi Mark!
>
> Thanks for trying my queries. I'm sorry (but not surprised!) to hear
> that they weren't any better than your original query.
>
> I think JENA-999 is really the key here - until it is implemented, I
> don't know any way of speeding this up. My (very fuzzy) understanding of
> ARQ is that it always performs joins with iterators, so there is no way
> to first retrieve both result sets (one from spatial, one from
> jena-text) and only then join them, unless you do the cross product and
> only limit with FILTER, which is even slower (options 2 and 3).
>
> By the way you didn't say which version of Jena you're using. Jena 3.0.1
> actually does include the initial JENA-999 patch which, despite its
> problems (related to retrieving score and especially the original
> literal - see JENA-1093), might actually work in your case. It was
> reverted only after the 3.0.1 release.
>
> Other than testing with 3.0.1, I'd suggest skipping jena-text for now
> and changing the text query part to a FILTER expression instead, with a
> REGEX or CONTAINS. Applying that kind of filter to 450 items shouldn't
> add much to the execution time of the spatial query.
>
> -Osma
>
> [1] https://issues.apache.org/jira/browse/JENA-999
> [2] https://issues.apache.org/jira/browse/JENA-1093
>
> On 21/12/15 13:02, Mark Wharton wrote:
>> Hi Osma.
>>
>> Thanks for your help. It was exactly the kind of help that I wanted.
>>
>> Haha, but life is rarely that simple...
>>
>> I ran all your versions and my original a couple of times each to see if
>> there was any difference in performance.
>>
>> query time1 time2
>> Orig 4.615 4.632
>> 1 4.648 4.608
>> 2 6.385 6.401
>> 3 6.353 6.442
>>
>> I think that proves that my "naive" version is working how I thought,
>> (and the same as your version 1). i.e. getting all the results from the
>> spatial predicate and then individually checking them against the text
>> predicate. (Which is why it's slower if you swap the predicates over,
>> as there are more matches on the text).
>>
>> This kind of query can't be that unusual, surely? Things with "foo" in
>> their text within 50Km of point "bar"?
>>
>> Mark
>>
>>
>> Technology Lead, Iotic Labs
>> +44 7973 674404
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 21/12/15 07:53, Osma Suominen wrote:
>>> Hi Mark!
>>>
>>> I'm not sure that the jena-external-index approach would help. It might
>>> or might not, depending on how it's implemented. AFAIK it's just an idea
>>> right now, I haven't seen any code.
>>>
>>> In any case I think the problem with jena-text and probably jena-spatial
>>> too (not very familiar with it) is that they are pretty fast when you
>>> can get by doing just a single query with an unbound subject. But when
>>> you instead have a fixed subject, or more likely a list of possible
>>> subjects, the performance will be very bad because every subject will be
>>> queried separately from the Lucene index. This is probably what happens
>>> with your query - the results of one query will be fed to the other.
>>> JENA-999 tried to address some of this by introducing a cache into
>>> jena-text, but the patch that was committed had other problems (it
>>> returned the wrong results in some cases) so it was reverted recently
>>> and an improved version hasn't yet been developed.
>>>
>>> In any case, you could try these variants of the original query:
>>> (I'm really on thin ground here, but I spent a couple of days last week
>>> trying to optimize a somewhat similar query involving jena-text and
>>> various other conditions, trying to find a solution with good
>>> performance)
>>>
>>> 1. Try to isolate the text and spatial queries with extra braces
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') }
>>> { (?ent ?score) text:query ('environment' 'lang:en') }
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>> 2. Use a FILTER expression to delay matching the results
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') .
>>> (?ent2 ?score) text:query ('environment' 'lang:en') .
>>> FILTER (?ent = ?ent2)
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>> 3. Combination of above ideas
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') }
>>> { (?ent2 ?score) text:query ('environment' 'lang:en') }
>>> FILTER (?ent = ?ent2)
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>>
>>> -Osma
>>>
>>>
>>> On 21/12/15 00:28, Mark Wharton wrote:
>>>> Hi Marco.
>>>>
>>>> Yes, that's it. The indexes work well in isolation, but don't combine
>>>> well. Smooshing them into a single index would be a great idea,
>>>> especially if the query could resolve both text and spatial predicates
>>>> with one matching scan of the index.
>>>>
>>>> Perhaps Stephen could be persuaded to pick up the pace on this one?
>>>>
>>>> Thanks Mark
>>>>
>>>> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann
>>>> <ma...@gmail.com> wrote:
>>>>> yes correct Mark I am only referring to the extra payload here for
>>>>> invoking the spatial filter in the SPARQL query.
>>>>>
>>>>> now that you mention a particular issue with the combined use of both
>>>>> jena-text and jena-spatial (something I am not aware of ) this might
>>>>> be related to duplicated code in the two projects. back in May Stephen
>>>>> Allen wrote on the dev-list that he is about to address some of this
>>>>> possibly in a new jena-external-index project.
>>>>>
>>>>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>>>>> <ma...@iotic-labs.com> wrote:
>>>>>> Hi
>>>>>>
>>>>>> Thanks for this. I've read the chapter in the book and now I'm not
>>>>> sure
>>>>>> if I misunderstand your reply or you've only addressed half of the
>>>>> problem.
>>>>>>
>>>>>> I'm not worried about the performance of the spatial search in
>>>>> isolation
>>>>>> - that's 97ms which is fine. The text search on its own takes a bit
>>>>>> longer but that's acceptable, too.
>>>>>>
>>>>>> It's when I put the spatial and text *together* that query time
>>>>> increase
>>>>>> by 10-30 times. That's the bit I don't understand and would like
>>>>> some
>>>>>> help with.
>>>>>>
>>>>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>>>>> than retrieving one set and looping through to retrieve the matches
>>>>>> individually on the other. (Which is my guess as to how it works).
>>>>>>
>>>>>> Thanks for your help so far.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> Technology Lead, Iotic Labs
>>>>>> mark.wharton@iotic-labs.com
>>>>>> https://www.iotic-labs.com
>>>>>>
>>>>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>>>>> it's a common spatial access method latency in paticular for small
>>>>>>> data sets. you can try a mbr range query instead.
>>>>>>>
>>>>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>>>>> by
>>>>>>> John Hebeler et. al.. 2009
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>>>>> <ma...@iotic-labs.com> wrote:
>>>>>>>> Hi Jena users.
>>>>>>>>
>>>>>>>> I'm having performance problems with a query that uses text and
>>>>> location
>>>>>>>> search
>>>>>>>>
>>>>>>>> The query is roughly this:
>>>>>>>>
>>>>>>>>
>>>>>>>> SELECT ?score ?ent
>>>>>>>> WHERE {
>>>>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>>>>> 70.01807880401611 'km') .
>>>>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> There are about 450 entities in that radius
>>>>>>>> There are about 2200 entities with environment in their
>>>>> rdfs:comment
>>>>>>>>
>>>>>>>> The query takes 5 seconds.
>>>>>>>>
>>>>>>>> I've tried this:
>>>>>>>> Commenting out the text predicate the query takes 97 ms
>>>>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>>>>
>>>>>>>>
>>>>>>>> My question is this... It looks like the query is separately
>>>>> getting
>>>>>>>> the results of the first two predicates and merging (somehow) to
>>>>> find
>>>>>>>> the intersection. Is there a formulation which will intersect the
>>>>> two
>>>>>>>> sets faster?
>>>>>>>>
>>>>>>>> Many TIAs,
>>>>>>>>
>>>>>>>> Mark
>>>>>>>> --
>>>>>>>> Technology Lead, Iotic Labs
>>>>>>>> mark.wharton@iotic-labs.com
>>>>>>>> https://www.iotic-labs.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> ---
>>>>> Marco Neumann
>>>>> KONA
>>>>
>>>
>>>
>
>
Re: Query with spatial *and* text searches.
Posted by Osma Suominen <os...@helsinki.fi>.
Hi Mark!
Thanks for trying my queries. I'm sorry (but not surprised!) to hear
that they weren't any better than your original query.
I think JENA-999 is really the key here - until it is implemented, I
don't know any way of speeding this up. My (very fuzzy) understanding of
ARQ is that it always performs joins with iterators, so there is no way
to first retrieve both result sets (one from spatial, one from
jena-text) and only then join them, unless you do the cross product and
only limit with FILTER, which is even slower (options 2 and 3).
By the way you didn't say which version of Jena you're using. Jena 3.0.1
actually does include the initial JENA-999 patch which, despite its
problems (related to retrieving score and especially the original
literal - see JENA-1093), might actually work in your case. It was
reverted only after the 3.0.1 release.
Other than testing with 3.0.1, I'd suggest skipping jena-text for now
and changing the text query part to a FILTER expression instead, with a
REGEX or CONTAINS. Applying that kind of filter to 450 items shouldn't
add much to the execution time of the spatial query.
-Osma
[1] https://issues.apache.org/jira/browse/JENA-999
[2] https://issues.apache.org/jira/browse/JENA-1093
On 21/12/15 13:02, Mark Wharton wrote:
> Hi Osma.
>
> Thanks for your help. It was exactly the kind of help that I wanted.
>
> Haha, but life is rarely that simple...
>
> I ran all your versions and my original a couple of times each to see if
> there was any difference in performance.
>
> query time1 time2
> Orig 4.615 4.632
> 1 4.648 4.608
> 2 6.385 6.401
> 3 6.353 6.442
>
> I think that proves that my "naive" version is working how I thought,
> (and the same as your version 1). i.e. getting all the results from the
> spatial predicate and then individually checking them against the text
> predicate. (Which is why it's slower if you swap the predicates over,
> as there are more matches on the text).
>
> This kind of query can't be that unusual, surely? Things with "foo" in
> their text within 50Km of point "bar"?
>
> Mark
>
>
> Technology Lead, Iotic Labs
> +44 7973 674404
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 21/12/15 07:53, Osma Suominen wrote:
>> Hi Mark!
>>
>> I'm not sure that the jena-external-index approach would help. It might
>> or might not, depending on how it's implemented. AFAIK it's just an idea
>> right now, I haven't seen any code.
>>
>> In any case I think the problem with jena-text and probably jena-spatial
>> too (not very familiar with it) is that they are pretty fast when you
>> can get by doing just a single query with an unbound subject. But when
>> you instead have a fixed subject, or more likely a list of possible
>> subjects, the performance will be very bad because every subject will be
>> queried separately from the Lucene index. This is probably what happens
>> with your query - the results of one query will be fed to the other.
>> JENA-999 tried to address some of this by introducing a cache into
>> jena-text, but the patch that was committed had other problems (it
>> returned the wrong results in some cases) so it was reverted recently
>> and an improved version hasn't yet been developed.
>>
>> In any case, you could try these variants of the original query:
>> (I'm really on thin ground here, but I spent a couple of days last week
>> trying to optimize a somewhat similar query involving jena-text and
>> various other conditions, trying to find a solution with good performance)
>>
>> 1. Try to isolate the text and spatial queries with extra braces
>>
>> SELECT ?score ?ent
>> WHERE {
>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.01807880401611 'km') }
>> { (?ent ?score) text:query ('environment' 'lang:en') }
>> ?ent rdf:type iotic:Entity .
>> }
>>
>> 2. Use a FILTER expression to delay matching the results
>>
>> SELECT ?score ?ent
>> WHERE {
>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.01807880401611 'km') .
>> (?ent2 ?score) text:query ('environment' 'lang:en') .
>> FILTER (?ent = ?ent2)
>> ?ent rdf:type iotic:Entity .
>> }
>>
>> 3. Combination of above ideas
>>
>> SELECT ?score ?ent
>> WHERE {
>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.01807880401611 'km') }
>> { (?ent2 ?score) text:query ('environment' 'lang:en') }
>> FILTER (?ent = ?ent2)
>> ?ent rdf:type iotic:Entity .
>> }
>>
>>
>> -Osma
>>
>>
>> On 21/12/15 00:28, Mark Wharton wrote:
>>> Hi Marco.
>>>
>>> Yes, that's it. The indexes work well in isolation, but don't combine
>>> well. Smooshing them into a single index would be a great idea,
>>> especially if the query could resolve both text and spatial predicates
>>> with one matching scan of the index.
>>>
>>> Perhaps Stephen could be persuaded to pick up the pace on this one?
>>>
>>> Thanks Mark
>>>
>>> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann
>>> <ma...@gmail.com> wrote:
>>>> yes correct Mark I am only referring to the extra payload here for
>>>> invoking the spatial filter in the SPARQL query.
>>>>
>>>> now that you mention a particular issue with the combined use of both
>>>> jena-text and jena-spatial (something I am not aware of ) this might
>>>> be related to duplicated code in the two projects. back in May Stephen
>>>> Allen wrote on the dev-list that he is about to address some of this
>>>> possibly in a new jena-external-index project.
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>>>>
>>>>
>>>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>>>> <ma...@iotic-labs.com> wrote:
>>>>> Hi
>>>>>
>>>>> Thanks for this. I've read the chapter in the book and now I'm not
>>>> sure
>>>>> if I misunderstand your reply or you've only addressed half of the
>>>> problem.
>>>>>
>>>>> I'm not worried about the performance of the spatial search in
>>>> isolation
>>>>> - that's 97ms which is fine. The text search on its own takes a bit
>>>>> longer but that's acceptable, too.
>>>>>
>>>>> It's when I put the spatial and text *together* that query time
>>>> increase
>>>>> by 10-30 times. That's the bit I don't understand and would like
>>>> some
>>>>> help with.
>>>>>
>>>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>>>> than retrieving one set and looping through to retrieve the matches
>>>>> individually on the other. (Which is my guess as to how it works).
>>>>>
>>>>> Thanks for your help so far.
>>>>>
>>>>> Mark
>>>>>
>>>>> Technology Lead, Iotic Labs
>>>>> mark.wharton@iotic-labs.com
>>>>> https://www.iotic-labs.com
>>>>>
>>>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>>>> it's a common spatial access method latency in paticular for small
>>>>>> data sets. you can try a mbr range query instead.
>>>>>>
>>>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>>>> by
>>>>>> John Hebeler et. al.. 2009
>>>>>>
>>>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>>>> <ma...@iotic-labs.com> wrote:
>>>>>>> Hi Jena users.
>>>>>>>
>>>>>>> I'm having performance problems with a query that uses text and
>>>> location
>>>>>>> search
>>>>>>>
>>>>>>> The query is roughly this:
>>>>>>>
>>>>>>>
>>>>>>> SELECT ?score ?ent
>>>>>>> WHERE {
>>>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>>>> 70.01807880401611 'km') .
>>>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> There are about 450 entities in that radius
>>>>>>> There are about 2200 entities with environment in their
>>>> rdfs:comment
>>>>>>>
>>>>>>> The query takes 5 seconds.
>>>>>>>
>>>>>>> I've tried this:
>>>>>>> Commenting out the text predicate the query takes 97 ms
>>>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>>>
>>>>>>>
>>>>>>> My question is this... It looks like the query is separately
>>>> getting
>>>>>>> the results of the first two predicates and merging (somehow) to
>>>> find
>>>>>>> the intersection. Is there a formulation which will intersect the
>>>> two
>>>>>>> sets faster?
>>>>>>>
>>>>>>> Many TIAs,
>>>>>>>
>>>>>>> Mark
>>>>>>> --
>>>>>>> Technology Lead, Iotic Labs
>>>>>>> mark.wharton@iotic-labs.com
>>>>>>> https://www.iotic-labs.com
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> ---
>>>> Marco Neumann
>>>> KONA
>>>
>>
>>
--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Kaikukatu 4)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.suominen@helsinki.fi
http://www.nationallibrary.fi
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi Osma.
Thanks for your help. It was exactly the kind of help that I wanted.
Haha, but life is rarely that simple...
I ran all your versions and my original a couple of times each to see if
there was any difference in performance.
query time1 time2
Orig 4.615 4.632
1 4.648 4.608
2 6.385 6.401
3 6.353 6.442
I think that proves that my "naive" version is working how I thought,
(and the same as your version 1). i.e. getting all the results from the
spatial predicate and then individually checking them against the text
predicate. (Which is why it's slower if you swap the predicates over,
as there are more matches on the text).
This kind of query can't be that unusual, surely? Things with "foo" in
their text within 50Km of point "bar"?
Mark
Technology Lead, Iotic Labs
+44 7973 674404
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 21/12/15 07:53, Osma Suominen wrote:
> Hi Mark!
>
> I'm not sure that the jena-external-index approach would help. It might
> or might not, depending on how it's implemented. AFAIK it's just an idea
> right now, I haven't seen any code.
>
> In any case I think the problem with jena-text and probably jena-spatial
> too (not very familiar with it) is that they are pretty fast when you
> can get by doing just a single query with an unbound subject. But when
> you instead have a fixed subject, or more likely a list of possible
> subjects, the performance will be very bad because every subject will be
> queried separately from the Lucene index. This is probably what happens
> with your query - the results of one query will be fed to the other.
> JENA-999 tried to address some of this by introducing a cache into
> jena-text, but the patch that was committed had other problems (it
> returned the wrong results in some cases) so it was reverted recently
> and an improved version hasn't yet been developed.
>
> In any case, you could try these variants of the original query:
> (I'm really on thin ground here, but I spent a couple of days last week
> trying to optimize a somewhat similar query involving jena-text and
> various other conditions, trying to find a solution with good performance)
>
> 1. Try to isolate the text and spatial queries with extra braces
>
> SELECT ?score ?ent
> WHERE {
> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.01807880401611 'km') }
> { (?ent ?score) text:query ('environment' 'lang:en') }
> ?ent rdf:type iotic:Entity .
> }
>
> 2. Use a FILTER expression to delay matching the results
>
> SELECT ?score ?ent
> WHERE {
> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.01807880401611 'km') .
> (?ent2 ?score) text:query ('environment' 'lang:en') .
> FILTER (?ent = ?ent2)
> ?ent rdf:type iotic:Entity .
> }
>
> 3. Combination of above ideas
>
> SELECT ?score ?ent
> WHERE {
> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.01807880401611 'km') }
> { (?ent2 ?score) text:query ('environment' 'lang:en') }
> FILTER (?ent = ?ent2)
> ?ent rdf:type iotic:Entity .
> }
>
>
> -Osma
>
>
> On 21/12/15 00:28, Mark Wharton wrote:
>> Hi Marco.
>>
>> Yes, that's it. The indexes work well in isolation, but don't combine
>> well. Smooshing them into a single index would be a great idea,
>> especially if the query could resolve both text and spatial predicates
>> with one matching scan of the index.
>>
>> Perhaps Stephen could be persuaded to pick up the pace on this one?
>>
>> Thanks Mark
>>
>> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann
>> <ma...@gmail.com> wrote:
>>> yes correct Mark I am only referring to the extra payload here for
>>> invoking the spatial filter in the SPARQL query.
>>>
>>> now that you mention a particular issue with the combined use of both
>>> jena-text and jena-spatial (something I am not aware of ) this might
>>> be related to duplicated code in the two projects. back in May Stephen
>>> Allen wrote on the dev-list that he is about to address some of this
>>> possibly in a new jena-external-index project.
>>>
>>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>>>
>>>
>>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>>> <ma...@iotic-labs.com> wrote:
>>>> Hi
>>>>
>>>> Thanks for this. I've read the chapter in the book and now I'm not
>>> sure
>>>> if I misunderstand your reply or you've only addressed half of the
>>> problem.
>>>>
>>>> I'm not worried about the performance of the spatial search in
>>> isolation
>>>> - that's 97ms which is fine. The text search on its own takes a bit
>>>> longer but that's acceptable, too.
>>>>
>>>> It's when I put the spatial and text *together* that query time
>>> increase
>>>> by 10-30 times. That's the bit I don't understand and would like
>>> some
>>>> help with.
>>>>
>>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>>> than retrieving one set and looping through to retrieve the matches
>>>> individually on the other. (Which is my guess as to how it works).
>>>>
>>>> Thanks for your help so far.
>>>>
>>>> Mark
>>>>
>>>> Technology Lead, Iotic Labs
>>>> mark.wharton@iotic-labs.com
>>>> https://www.iotic-labs.com
>>>>
>>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>>> it's a common spatial access method latency in paticular for small
>>>>> data sets. you can try a mbr range query instead.
>>>>>
>>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>>> by
>>>>> John Hebeler et. al.. 2009
>>>>>
>>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>>> <ma...@iotic-labs.com> wrote:
>>>>>> Hi Jena users.
>>>>>>
>>>>>> I'm having performance problems with a query that uses text and
>>> location
>>>>>> search
>>>>>>
>>>>>> The query is roughly this:
>>>>>>
>>>>>>
>>>>>> SELECT ?score ?ent
>>>>>> WHERE {
>>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>>> 70.01807880401611 'km') .
>>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>>> ?ent rdf:type iotic:Entity .
>>>>>> }
>>>>>>
>>>>>>
>>>>>> There are about 450 entities in that radius
>>>>>> There are about 2200 entities with environment in their
>>> rdfs:comment
>>>>>>
>>>>>> The query takes 5 seconds.
>>>>>>
>>>>>> I've tried this:
>>>>>> Commenting out the text predicate the query takes 97 ms
>>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>>
>>>>>>
>>>>>> My question is this... It looks like the query is separately
>>> getting
>>>>>> the results of the first two predicates and merging (somehow) to
>>> find
>>>>>> the intersection. Is there a formulation which will intersect the
>>> two
>>>>>> sets faster?
>>>>>>
>>>>>> Many TIAs,
>>>>>>
>>>>>> Mark
>>>>>> --
>>>>>> Technology Lead, Iotic Labs
>>>>>> mark.wharton@iotic-labs.com
>>>>>> https://www.iotic-labs.com
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> ---
>>> Marco Neumann
>>> KONA
>>
>
>
Re: Query with spatial *and* text searches.
Posted by Osma Suominen <os...@helsinki.fi>.
Hi Mark!
I'm not sure that the jena-external-index approach would help. It might
or might not, depending on how it's implemented. AFAIK it's just an idea
right now, I haven't seen any code.
In any case I think the problem with jena-text and probably jena-spatial
too (not very familiar with it) is that they are pretty fast when you
can get by doing just a single query with an unbound subject. But when
you instead have a fixed subject, or more likely a list of possible
subjects, the performance will be very bad because every subject will be
queried separately from the Lucene index. This is probably what happens
with your query - the results of one query will be fed to the other.
JENA-999 tried to address some of this by introducing a cache into
jena-text, but the patch that was committed had other problems (it
returned the wrong results in some cases) so it was reverted recently
and an improved version hasn't yet been developed.
In any case, you could try these variants of the original query:
(I'm really on thin ground here, but I spent a couple of days last week
trying to optimize a somewhat similar query involving jena-text and
various other conditions, trying to find a solution with good performance)
1. Try to isolate the text and spatial queries with extra braces
SELECT ?score ?ent
WHERE {
{ ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.01807880401611 'km') }
{ (?ent ?score) text:query ('environment' 'lang:en') }
?ent rdf:type iotic:Entity .
}
2. Use a FILTER expression to delay matching the results
SELECT ?score ?ent
WHERE {
?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.01807880401611 'km') .
(?ent2 ?score) text:query ('environment' 'lang:en') .
FILTER (?ent = ?ent2)
?ent rdf:type iotic:Entity .
}
3. Combination of above ideas
SELECT ?score ?ent
WHERE {
{ ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.01807880401611 'km') }
{ (?ent2 ?score) text:query ('environment' 'lang:en') }
FILTER (?ent = ?ent2)
?ent rdf:type iotic:Entity .
}
-Osma
On 21/12/15 00:28, Mark Wharton wrote:
> Hi Marco.
>
> Yes, that's it. The indexes work well in isolation, but don't combine well. Smooshing them into a single index would be a great idea, especially if the query could resolve both text and spatial predicates with one matching scan of the index.
>
> Perhaps Stephen could be persuaded to pick up the pace on this one?
>
> Thanks Mark
>
> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann <ma...@gmail.com> wrote:
>> yes correct Mark I am only referring to the extra payload here for
>> invoking the spatial filter in the SPARQL query.
>>
>> now that you mention a particular issue with the combined use of both
>> jena-text and jena-spatial (something I am not aware of ) this might
>> be related to duplicated code in the two projects. back in May Stephen
>> Allen wrote on the dev-list that he is about to address some of this
>> possibly in a new jena-external-index project.
>>
>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>>
>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>> <ma...@iotic-labs.com> wrote:
>>> Hi
>>>
>>> Thanks for this. I've read the chapter in the book and now I'm not
>> sure
>>> if I misunderstand your reply or you've only addressed half of the
>> problem.
>>>
>>> I'm not worried about the performance of the spatial search in
>> isolation
>>> - that's 97ms which is fine. The text search on its own takes a bit
>>> longer but that's acceptable, too.
>>>
>>> It's when I put the spatial and text *together* that query time
>> increase
>>> by 10-30 times. That's the bit I don't understand and would like
>> some
>>> help with.
>>>
>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>> than retrieving one set and looping through to retrieve the matches
>>> individually on the other. (Which is my guess as to how it works).
>>>
>>> Thanks for your help so far.
>>>
>>> Mark
>>>
>>> Technology Lead, Iotic Labs
>>> mark.wharton@iotic-labs.com
>>> https://www.iotic-labs.com
>>>
>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>> it's a common spatial access method latency in paticular for small
>>>> data sets. you can try a mbr range query instead.
>>>>
>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>> by
>>>> John Hebeler et. al.. 2009
>>>>
>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>> <ma...@iotic-labs.com> wrote:
>>>>> Hi Jena users.
>>>>>
>>>>> I'm having performance problems with a query that uses text and
>> location
>>>>> search
>>>>>
>>>>> The query is roughly this:
>>>>>
>>>>>
>>>>> SELECT ?score ?ent
>>>>> WHERE {
>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>> 70.01807880401611 'km') .
>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>> ?ent rdf:type iotic:Entity .
>>>>> }
>>>>>
>>>>>
>>>>> There are about 450 entities in that radius
>>>>> There are about 2200 entities with environment in their
>> rdfs:comment
>>>>>
>>>>> The query takes 5 seconds.
>>>>>
>>>>> I've tried this:
>>>>> Commenting out the text predicate the query takes 97 ms
>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>
>>>>>
>>>>> My question is this... It looks like the query is separately
>> getting
>>>>> the results of the first two predicates and merging (somehow) to
>> find
>>>>> the intersection. Is there a formulation which will intersect the
>> two
>>>>> sets faster?
>>>>>
>>>>> Many TIAs,
>>>>>
>>>>> Mark
>>>>> --
>>>>> Technology Lead, Iotic Labs
>>>>> mark.wharton@iotic-labs.com
>>>>> https://www.iotic-labs.com
>>>>
>>>>
>>>>
>>
>>
>>
>> --
>>
>>
>> ---
>> Marco Neumann
>> KONA
>
--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Kaikukatu 4)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.suominen@helsinki.fi
http://www.nationallibrary.fi
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi Marco.
Yes, that's it. The indexes work well in isolation, but don't combine well. Smooshing them into a single index would be a great idea, especially if the query could resolve both text and spatial predicates with one matching scan of the index.
Perhaps Stephen could be persuaded to pick up the pace on this one?
Thanks Mark
On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann <ma...@gmail.com> wrote:
>yes correct Mark I am only referring to the extra payload here for
>invoking the spatial filter in the SPARQL query.
>
>now that you mention a particular issue with the combined use of both
>jena-text and jena-spatial (something I am not aware of ) this might
>be related to duplicated code in the two projects. back in May Stephen
>Allen wrote on the dev-list that he is about to address some of this
>possibly in a new jena-external-index project.
>
>http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
>
>On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
><ma...@iotic-labs.com> wrote:
>> Hi
>>
>> Thanks for this. I've read the chapter in the book and now I'm not
>sure
>> if I misunderstand your reply or you've only addressed half of the
>problem.
>>
>> I'm not worried about the performance of the spatial search in
>isolation
>> - that's 97ms which is fine. The text search on its own takes a bit
>> longer but that's acceptable, too.
>>
>> It's when I put the spatial and text *together* that query time
>increase
>> by 10-30 times. That's the bit I don't understand and would like
>some
>> help with.
>>
>> Is there a SPARQL query formulation that can "AND the indexes" rather
>> than retrieving one set and looping through to retrieve the matches
>> individually on the other. (Which is my guess as to how it works).
>>
>> Thanks for your help so far.
>>
>> Mark
>>
>> Technology Lead, Iotic Labs
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 18/12/15 18:59, Marco Neumann wrote:
>>> it's a common spatial access method latency in paticular for small
>>> data sets. you can try a mbr range query instead.
>>>
>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>by
>>> John Hebeler et. al.. 2009
>>>
>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>> <ma...@iotic-labs.com> wrote:
>>>> Hi Jena users.
>>>>
>>>> I'm having performance problems with a query that uses text and
>location
>>>> search
>>>>
>>>> The query is roughly this:
>>>>
>>>>
>>>> SELECT ?score ?ent
>>>> WHERE {
>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>> 70.01807880401611 'km') .
>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>> ?ent rdf:type iotic:Entity .
>>>> }
>>>>
>>>>
>>>> There are about 450 entities in that radius
>>>> There are about 2200 entities with environment in their
>rdfs:comment
>>>>
>>>> The query takes 5 seconds.
>>>>
>>>> I've tried this:
>>>> Commenting out the text predicate the query takes 97 ms
>>>> Commenting out the spatial predicate the query takes 438 ms
>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>
>>>>
>>>> My question is this... It looks like the query is separately
>getting
>>>> the results of the first two predicates and merging (somehow) to
>find
>>>> the intersection. Is there a formulation which will intersect the
>two
>>>> sets faster?
>>>>
>>>> Many TIAs,
>>>>
>>>> Mark
>>>> --
>>>> Technology Lead, Iotic Labs
>>>> mark.wharton@iotic-labs.com
>>>> https://www.iotic-labs.com
>>>
>>>
>>>
>
>
>
>--
>
>
>---
>Marco Neumann
>KONA
Re: Query with spatial *and* text searches.
Posted by Marco Neumann <ma...@gmail.com>.
yes correct Mark I am only referring to the extra payload here for
invoking the spatial filter in the SPARQL query.
now that you mention a particular issue with the combined use of both
jena-text and jena-spatial (something I am not aware of ) this might
be related to duplicated code in the two projects. back in May Stephen
Allen wrote on the dev-list that he is about to address some of this
possibly in a new jena-external-index project.
http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3CCAPTxtVPwu2iJOgYj0kX8O6-07YoKk5g1t32B_K3g_cjaQVKZ+w@mail.gmail.com%3E
On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
<ma...@iotic-labs.com> wrote:
> Hi
>
> Thanks for this. I've read the chapter in the book and now I'm not sure
> if I misunderstand your reply or you've only addressed half of the problem.
>
> I'm not worried about the performance of the spatial search in isolation
> - that's 97ms which is fine. The text search on its own takes a bit
> longer but that's acceptable, too.
>
> It's when I put the spatial and text *together* that query time increase
> by 10-30 times. That's the bit I don't understand and would like some
> help with.
>
> Is there a SPARQL query formulation that can "AND the indexes" rather
> than retrieving one set and looping through to retrieve the matches
> individually on the other. (Which is my guess as to how it works).
>
> Thanks for your help so far.
>
> Mark
>
> Technology Lead, Iotic Labs
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
>
> On 18/12/15 18:59, Marco Neumann wrote:
>> it's a common spatial access method latency in paticular for small
>> data sets. you can try a mbr range query instead.
>>
>> see Chapter 13 Managing Space and Time in Semantic Web Programming by
>> John Hebeler et. al.. 2009
>>
>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>> <ma...@iotic-labs.com> wrote:
>>> Hi Jena users.
>>>
>>> I'm having performance problems with a query that uses text and location
>>> search
>>>
>>> The query is roughly this:
>>>
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') .
>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>>
>>> There are about 450 entities in that radius
>>> There are about 2200 entities with environment in their rdfs:comment
>>>
>>> The query takes 5 seconds.
>>>
>>> I've tried this:
>>> Commenting out the text predicate the query takes 97 ms
>>> Commenting out the spatial predicate the query takes 438 ms
>>> Swapping the spatial and text predicates it takes 15 seconds
>>>
>>>
>>> My question is this... It looks like the query is separately getting
>>> the results of the first two predicates and merging (somehow) to find
>>> the intersection. Is there a formulation which will intersect the two
>>> sets faster?
>>>
>>> Many TIAs,
>>>
>>> Mark
>>> --
>>> Technology Lead, Iotic Labs
>>> mark.wharton@iotic-labs.com
>>> https://www.iotic-labs.com
>>
>>
>>
--
---
Marco Neumann
KONA
Re: Query with spatial *and* text searches.
Posted by Mark Wharton <ma...@iotic-labs.com>.
Hi
Thanks for this. I've read the chapter in the book and now I'm not sure
if I misunderstand your reply or you've only addressed half of the problem.
I'm not worried about the performance of the spatial search in isolation
- that's 97ms which is fine. The text search on its own takes a bit
longer but that's acceptable, too.
It's when I put the spatial and text *together* that query time increase
by 10-30 times. That's the bit I don't understand and would like some
help with.
Is there a SPARQL query formulation that can "AND the indexes" rather
than retrieving one set and looping through to retrieve the matches
individually on the other. (Which is my guess as to how it works).
Thanks for your help so far.
Mark
Technology Lead, Iotic Labs
mark.wharton@iotic-labs.com
https://www.iotic-labs.com
On 18/12/15 18:59, Marco Neumann wrote:
> it's a common spatial access method latency in paticular for small
> data sets. you can try a mbr range query instead.
>
> see Chapter 13 Managing Space and Time in Semantic Web Programming by
> John Hebeler et. al.. 2009
>
> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
> <ma...@iotic-labs.com> wrote:
>> Hi Jena users.
>>
>> I'm having performance problems with a query that uses text and location
>> search
>>
>> The query is roughly this:
>>
>>
>> SELECT ?score ?ent
>> WHERE {
>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>> 70.01807880401611 'km') .
>> (?ent ?score) text:query ('environment' 'lang:en') .
>> ?ent rdf:type iotic:Entity .
>> }
>>
>>
>> There are about 450 entities in that radius
>> There are about 2200 entities with environment in their rdfs:comment
>>
>> The query takes 5 seconds.
>>
>> I've tried this:
>> Commenting out the text predicate the query takes 97 ms
>> Commenting out the spatial predicate the query takes 438 ms
>> Swapping the spatial and text predicates it takes 15 seconds
>>
>>
>> My question is this... It looks like the query is separately getting
>> the results of the first two predicates and merging (somehow) to find
>> the intersection. Is there a formulation which will intersect the two
>> sets faster?
>>
>> Many TIAs,
>>
>> Mark
>> --
>> Technology Lead, Iotic Labs
>> mark.wharton@iotic-labs.com
>> https://www.iotic-labs.com
>
>
>
Re: Query with spatial *and* text searches.
Posted by Marco Neumann <ma...@gmail.com>.
it's a common spatial access method latency in paticular for small
data sets. you can try a mbr range query instead.
see Chapter 13 Managing Space and Time in Semantic Web Programming by
John Hebeler et. al.. 2009
On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
<ma...@iotic-labs.com> wrote:
> Hi Jena users.
>
> I'm having performance problems with a query that uses text and location
> search
>
> The query is roughly this:
>
>
> SELECT ?score ?ent
> WHERE {
> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
> 70.01807880401611 'km') .
> (?ent ?score) text:query ('environment' 'lang:en') .
> ?ent rdf:type iotic:Entity .
> }
>
>
> There are about 450 entities in that radius
> There are about 2200 entities with environment in their rdfs:comment
>
> The query takes 5 seconds.
>
> I've tried this:
> Commenting out the text predicate the query takes 97 ms
> Commenting out the spatial predicate the query takes 438 ms
> Swapping the spatial and text predicates it takes 15 seconds
>
>
> My question is this... It looks like the query is separately getting
> the results of the first two predicates and merging (somehow) to find
> the intersection. Is there a formulation which will intersect the two
> sets faster?
>
> Many TIAs,
>
> Mark
> --
> Technology Lead, Iotic Labs
> mark.wharton@iotic-labs.com
> https://www.iotic-labs.com
--
---
Marco Neumann
KONA