You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Piotr Nowara <pi...@gmail.com> on 2020/09/10 11:16:07 UTC
Jena performance optimization
Dear Community,
could you point me to resources on Jena performance optimization?
I found this excellent presentation [1] covering a lot of interesting
topics but I'm not sure how up-to-date it might be. I also read the Jena
docs which gave me some hints but also created some confusion. My main
areas of interest are:
1. How to make complex INSERT WHERE statements faster? Does statistic
based optimization work for them? Why is there no --explain function for
them? Would CONSTRUCTs be faster?
2. optimization methods (fixed vs. stats) - which is a default one? how
to run tdbstats against running Fuseki server? how to run tdbstats on a
database with triples in multiple named graphs (I have to use separate
named graphs and I can't use default union graph)
Thanks,
Piotr
[1]
https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
Re: Jena performance optimization
Posted by Andy Seaborne <an...@apache.org>.
On 11/09/2020 17:46, Piotr Nowara wrote:
> Hi Andy,
>
> thanks for your explanations. Could you clarify what you mean by "BGP part"
> and that "the optimizer is only for BGP anyway"? Does it mean stats affect
> only matching triple/quad patterns against Jena indexes or there is
> something more to it I'm not aware of?
The optimizer has two parts.
It rewrites the algebra. Some important transformations are placing
filters and deciding what join algorithm to use to combine parts of the
query. This stage can break up BGPs (for filters) but the reordering
happen in the next stage in a "stage generator" if still triples.
If it's been rewritten to quads, reordering is done in the OpExecutior
for the quads execution. Similar process, organised slightly differently.
When a BGP is executed, the triples are reordered according to the BGP
transform - fixed or stats. This is storage type sensitive. Even
in-memory gets some reordering, which is the "fixed" algorithm.
The practical experience is that the "fixed" algorithm does the job
comparable to the stats for the majority of deployments. Only unusual
shaped data shows much benefit of the stats in practice (which is good
from a simplicity point of view and bad from looking at the
implementation done to provide stats!)
Both the fixed and stats algorithms do not reorder triple patterns that
the algorithm considers to have the same weight.
In SPARQL, BGP's are an extension point for inference. I don't know if
that is used much nowadays. But it is also a way to plugging in custom
storage without having to implement more of the SPARQL execution so
either as a lower footprint method or on the way to a deeper integration
but still have something to run, that option is still supported.
The end-to-end process has grown over time. There is a good case for
stripping out unused extensions points, and lessening the distinction of
algebra and BGP optimization steps.
Andy
>
> Thanks,
> Piotr
>
> pt., 11 wrz 2020 o 16:01 Andy Seaborne <an...@apache.org> napisał(a):
>
>>
>>
>> On 10/09/2020 12:16, Piotr Nowara wrote:
>>> Dear Community,
>>>
>>> could you point me to resources on Jena performance optimization?
>>>
>>> I found this excellent presentation [1] covering a lot of interesting
>>> topics but I'm not sure how up-to-date it might be. I also read the Jena
>>> docs which gave me some hints but also created some confusion. My main
>>> areas of interest are:
>>>
>>> 1. How to make complex INSERT WHERE statements faster?
>>
>> Usually, by reordering/rewriting the WHERE clause based on knowing the
>> data better than the system.
>>
>> The stats optimizer is only for BGPs anyway - the transformation of the
>> alegbra is done separately.
>>
>>> Does statistic
>>> based optimization work for them?
>>
>> Yes.
>>
>>> Why is there no --explain function for them?
>>
>> The WHERE part is just a query so extract it, add "SELECT *" and use the
>> query tools.
>>
>>> Would CONSTRUCTs be faster?
>> No.
>>
>>> 2. optimization methods (fixed vs. stats) - which is a default one?
>> Fixed.
>>
>>> how
>>> to run tdbstats against running Fuseki server?
>> I'm afraid you can't.
>>
>>> how to run tdbstats on a
>>> database with triples in multiple named graphs (I have to use
>> separate
>>> named graphs and I can't use default union graph)
>>
>> Are you sure it is the BGP part that is slow?
>>
>> Andy
>>
>>>
>>>
>>> Thanks,
>>> Piotr
>>>
>>>
>>> [1]
>>>
>> https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
>>>
>>
>
Re: Jena performance optimization
Posted by Piotr Nowara <pi...@gmail.com>.
Hi Andy,
thanks for your explanations. Could you clarify what you mean by "BGP part"
and that "the optimizer is only for BGP anyway"? Does it mean stats affect
only matching triple/quad patterns against Jena indexes or there is
something more to it I'm not aware of?
Thanks,
Piotr
pt., 11 wrz 2020 o 16:01 Andy Seaborne <an...@apache.org> napisał(a):
>
>
> On 10/09/2020 12:16, Piotr Nowara wrote:
> > Dear Community,
> >
> > could you point me to resources on Jena performance optimization?
> >
> > I found this excellent presentation [1] covering a lot of interesting
> > topics but I'm not sure how up-to-date it might be. I also read the Jena
> > docs which gave me some hints but also created some confusion. My main
> > areas of interest are:
> >
> > 1. How to make complex INSERT WHERE statements faster?
>
> Usually, by reordering/rewriting the WHERE clause based on knowing the
> data better than the system.
>
> The stats optimizer is only for BGPs anyway - the transformation of the
> alegbra is done separately.
>
> > Does statistic
> > based optimization work for them?
>
> Yes.
>
> > Why is there no --explain function for them?
>
> The WHERE part is just a query so extract it, add "SELECT *" and use the
> query tools.
>
> > Would CONSTRUCTs be faster?
> No.
>
> > 2. optimization methods (fixed vs. stats) - which is a default one?
> Fixed.
>
> > how
> > to run tdbstats against running Fuseki server?
> I'm afraid you can't.
>
> > how to run tdbstats on a
> > database with triples in multiple named graphs (I have to use
> separate
> > named graphs and I can't use default union graph)
>
> Are you sure it is the BGP part that is slow?
>
> Andy
>
> >
> >
> > Thanks,
> > Piotr
> >
> >
> > [1]
> >
> https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
> >
>
Re: Jena performance optimization
Posted by Andy Seaborne <an...@apache.org>.
On 10/09/2020 12:16, Piotr Nowara wrote:
> Dear Community,
>
> could you point me to resources on Jena performance optimization?
>
> I found this excellent presentation [1] covering a lot of interesting
> topics but I'm not sure how up-to-date it might be. I also read the Jena
> docs which gave me some hints but also created some confusion. My main
> areas of interest are:
>
> 1. How to make complex INSERT WHERE statements faster?
Usually, by reordering/rewriting the WHERE clause based on knowing the
data better than the system.
The stats optimizer is only for BGPs anyway - the transformation of the
alegbra is done separately.
> Does statistic
> based optimization work for them?
Yes.
> Why is there no --explain function for them?
The WHERE part is just a query so extract it, add "SELECT *" and use the
query tools.
> Would CONSTRUCTs be faster?
No.
> 2. optimization methods (fixed vs. stats) - which is a default one?
Fixed.
> how
> to run tdbstats against running Fuseki server?
I'm afraid you can't.
> how to run tdbstats on a
> database with triples in multiple named graphs (I have to use separate
> named graphs and I can't use default union graph)
Are you sure it is the BGP part that is slow?
Andy
>
>
> Thanks,
> Piotr
>
>
> [1]
> https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
>