You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Élie Roux <el...@telecom-bretagne.eu> on 2020/09/02 13:41:58 UTC
Re: shortcut for querying dates fast?
> This is what was discovered before - the cost of scanning and filtering
> isn't that high and why outlier cases may be measurable faster, the bulk
> of queries will be marginally faster. There is always a lot of things
> that can be done; it comes down to contributions and priorities.
>
> And the cost of the join? and of the CONSTRUCT? And if Fuseki, the HTTP
> costs which vary from trivial to a lot depending on result sizes.
> connection caching. The point is "it is complicated" and that means
> however good the point improvement is, it may not have a significant
> overall benefit.
>
> Investigation needed before jumping into implementation.
Well... I don't disagree that it is complicated. I now have a
relatively straightforward query that takes 1.7s even after a few
attempts:
select (count (distinct ?e) as ?count) where {
{
?le adm:logDate ?sdate .
FILTER(?sdate > "2020-08-20T00:00:00"^^xsd:dateTime)
?le a adm:Synced .
?va adm:logEntry ?le ;
adm:adminAbout ?v .
?v bdo:volumeOf ?e .
}
}
I'm reading the 1.7s in the Fuseki logs. Interestingly if I take a
value in the future for the date and get 0 result, the query still
takes 1.7s, for instance:
select ?le where {
{
?le adm:logDate ?sdate .
FILTER(?sdate > "2020-10-20T00:00:00"^^xsd:dateTime)
}
}
So it's a bit hard for me to think the bottleneck could be
elsewhere... what other possible bottleneck should I look at?
Note that this contradicts previous findings where a similar query was
faster (around 300ms) if the indexes were not cold... but oddly I
can't reproduce it anymore, the 1.7s result has been consistent over
many queries in a short period of time, so the indexes were not
cold...
I understand it's a lot of code to write and that it's a big project, sorry.
> ARQ does not use the Model API. It's an extension to the ARQ algebra,
> OpExecutor, and subclasses, and one or more optimization Transforms to
> detect the case in a query.
>
> Overall, this isn't an API issue - it's the cost of implementing that
> API vs not doing something elsewhere.
Ok yes
Best,
--
Elie
Re: shortcut for querying dates fast?
Posted by Andy Seaborne <an...@apache.org>.
On 04/09/2020 14:38, Élie Roux wrote:
>> select ?le ?la where {
>> ?le adm:logDate ?sdate .
>> FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
>> BIND(?le AS ?la)
>> }
>
> In that trivial case yes, the longer query doesn't allow that:
Put the common part in a single place:
WHERE {
... common part ...
{
...
} UNION {
...
}
}
and reorder to put common part together.
>
> construct {
> ?eadm tmp:lastSync ?d .
> ?eadm tmp:dateCreated ?sdate .
> }
> WHERE
> {
> {
> ?le adm:logDate ?sdate .
> FILTER(?sdate > "2020-08-20T00:00:00"^^xsd:dateTime)
> ?le a adm:Synced .
> ?va adm:logEntry ?le ;
> adm:adminAbout ?v .
> ?v bdo:volumeOf ?iinstance .
> ?eadm adm:adminAbout ?iinstance .
> } union {
> ?le adm:logDate ?sdate .
> FILTER(?sdate > "2020-08-20T00:00:00"^^xsd:dateTime)
> ?le a adm:InitialDataCreation .
> ?eadm adm:logEntry ?le ;
> adm:adminAbout ?e .
> ?e a ?type .
> FILTER (?type != bdo:ImageGroup)
> }
> }
>
> and takes 3.5s.
>
> Best,
>
Re: shortcut for querying dates fast?
Posted by Élie Roux <el...@telecom-bretagne.eu>.
> select ?le ?la where {
> ?le adm:logDate ?sdate .
> FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
> BIND(?le AS ?la)
> }
In that trivial case yes, the longer query doesn't allow that:
construct {
?eadm tmp:lastSync ?d .
?eadm tmp:dateCreated ?sdate .
}
WHERE
{
{
?le adm:logDate ?sdate .
FILTER(?sdate > "2020-08-20T00:00:00"^^xsd:dateTime)
?le a adm:Synced .
?va adm:logEntry ?le ;
adm:adminAbout ?v .
?v bdo:volumeOf ?iinstance .
?eadm adm:adminAbout ?iinstance .
} union {
?le adm:logDate ?sdate .
FILTER(?sdate > "2020-08-20T00:00:00"^^xsd:dateTime)
?le a adm:InitialDataCreation .
?eadm adm:logEntry ?le ;
adm:adminAbout ?e .
?e a ?type .
FILTER (?type != bdo:ImageGroup)
}
}
and takes 3.5s.
Best,
--
Elie
Re: shortcut for querying dates fast?
Posted by Andy Seaborne <an...@apache.org>.
On 02/09/2020 16:17, Élie Roux wrote:
> P.S.: Here's another aspect of the problem (although in a different
> aspect of the code). If I make the same query + filter on dates in
> unions, there seems to be no cache mechanism, as the following takes
> 3.5s instead of 1.7s:
>
> select ?le ?Nla where {
> {
> ?le adm:logDate ?sdate .
> FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
> } union {
> ?la adm:logDate ?sdate .
> FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
> }
> }
select ?le ?la where {
?le adm:logDate ?sdate .
FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
BIND(?le AS ?la)
}
>
> perhaps that could be the subject of another issue? Or is it another
> case that is too abnormal to be optimized? (In a real-life use case, I
> would but two different BGP after each FILTER of course)
>
> Best,
>
Re: shortcut for querying dates fast?
Posted by Élie Roux <el...@telecom-bretagne.eu>.
P.S.: Here's another aspect of the problem (although in a different
aspect of the code). If I make the same query + filter on dates in
unions, there seems to be no cache mechanism, as the following takes
3.5s instead of 1.7s:
select ?le ?Nla where {
{
?le adm:logDate ?sdate .
FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
} union {
?la adm:logDate ?sdate .
FILTER(?sdate > "2021-08-20T00:00:00"^^xsd:dateTime)
}
}
perhaps that could be the subject of another issue? Or is it another
case that is too abnormal to be optimized? (In a real-life use case, I
would but two different BGP after each FILTER of course)
Best,
--
Elie