You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Piotr Nowara <pi...@gmail.com> on 2020/09/10 11:16:07 UTC

Jena performance optimization

Dear Community,

could you point me to resources on Jena performance optimization?

I found this excellent presentation [1] covering a lot of interesting
topics but I'm not sure how up-to-date it might be. I also read the Jena
docs which gave me some hints but also created some confusion. My main
areas of interest are:

   1. How to make complex INSERT WHERE statements faster? Does statistic
   based optimization work for them? Why is there no --explain function for
   them? Would CONSTRUCTs be faster?
   2. optimization methods (fixed vs. stats) - which is a default one? how
   to run tdbstats against running Fuseki server? how to run tdbstats on a
   database with triples in multiple named graphs (I have to use separate
   named graphs and I can't use default union graph)


Thanks,
Piotr


[1]
https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf

Re: Jena performance optimization

Posted by Andy Seaborne <an...@apache.org>.

On 11/09/2020 17:46, Piotr Nowara wrote:
> Hi Andy,
> 
> thanks for your explanations. Could you clarify what you mean by "BGP part"
> and that "the optimizer is only for BGP anyway"? Does it mean stats affect
> only matching triple/quad patterns against Jena indexes or there is
> something more to it I'm not aware of?

The optimizer has two parts.

It rewrites the algebra. Some important transformations are placing 
filters and deciding what join algorithm to use to combine parts of the 
query.  This stage can break up BGPs (for filters) but the reordering 
happen in the next stage in a "stage generator" if still triples.

If it's been rewritten to quads, reordering is done in the OpExecutior 
for the quads execution. Similar process, organised slightly differently.

When a BGP is executed, the triples are reordered according to the BGP 
transform - fixed or stats. This is storage type sensitive.  Even 
in-memory gets some reordering, which is the "fixed" algorithm.

The practical experience is that the "fixed" algorithm does the job 
comparable to the stats for the majority of deployments.  Only unusual 
shaped data shows much benefit of the stats in practice (which is good 
from a simplicity point of view and bad from looking at the 
implementation done to provide stats!)

Both the fixed and stats algorithms do not reorder triple patterns that 
the algorithm considers to have the same weight.

In SPARQL, BGP's are an extension point for inference. I don't know if 
that is used much nowadays. But it is also a way to plugging in custom 
storage without having to implement more of the SPARQL execution so 
either as a lower footprint method or on the way to a deeper integration 
but still have something to run, that option is still supported.

The end-to-end process has grown over time. There is a good case for 
stripping out unused extensions points, and lessening the distinction of 
algebra and BGP optimization steps.

     Andy

> 
> Thanks,
> Piotr
> 
> pt., 11 wrz 2020 o 16:01 Andy Seaborne <an...@apache.org> napisał(a):
> 
>>
>>
>> On 10/09/2020 12:16, Piotr Nowara wrote:
>>> Dear Community,
>>>
>>> could you point me to resources on Jena performance optimization?
>>>
>>> I found this excellent presentation [1] covering a lot of interesting
>>> topics but I'm not sure how up-to-date it might be. I also read the Jena
>>> docs which gave me some hints but also created some confusion. My main
>>> areas of interest are:
>>>
>>>      1. How to make complex INSERT WHERE statements faster?
>>
>> Usually, by reordering/rewriting the WHERE clause based on knowing the
>> data better than the system.
>>
>> The stats optimizer is only for BGPs anyway - the transformation of the
>> alegbra is done separately.
>>
>>> Does statistic
>>>      based optimization work for them?
>>
>> Yes.
>>
>>> Why is there no --explain function for them?
>>
>> The WHERE part is just a query so extract it, add "SELECT *" and use the
>> query tools.
>>
>>> Would CONSTRUCTs be faster?
>> No.
>>
>>>      2. optimization methods (fixed vs. stats) - which is a default one?
>> Fixed.
>>
>>> how
>>>      to run tdbstats against running Fuseki server?
>> I'm afraid you can't.
>>
>>> how to run tdbstats on a
>>>      database with triples in multiple named graphs (I have to use
>> separate
>>>      named graphs and I can't use default union graph)
>>
>> Are you sure it is the BGP part that is slow?
>>
>>       Andy
>>
>>>
>>>
>>> Thanks,
>>> Piotr
>>>
>>>
>>> [1]
>>>
>> https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
>>>
>>
> 

Re: Jena performance optimization

Posted by Piotr Nowara <pi...@gmail.com>.
Hi Andy,

thanks for your explanations. Could you clarify what you mean by "BGP part"
and that "the optimizer is only for BGP anyway"? Does it mean stats affect
only matching triple/quad patterns against Jena indexes or there is
something more to it I'm not aware of?

Thanks,
Piotr

pt., 11 wrz 2020 o 16:01 Andy Seaborne <an...@apache.org> napisał(a):

>
>
> On 10/09/2020 12:16, Piotr Nowara wrote:
> > Dear Community,
> >
> > could you point me to resources on Jena performance optimization?
> >
> > I found this excellent presentation [1] covering a lot of interesting
> > topics but I'm not sure how up-to-date it might be. I also read the Jena
> > docs which gave me some hints but also created some confusion. My main
> > areas of interest are:
> >
> >     1. How to make complex INSERT WHERE statements faster?
>
> Usually, by reordering/rewriting the WHERE clause based on knowing the
> data better than the system.
>
> The stats optimizer is only for BGPs anyway - the transformation of the
> alegbra is done separately.
>
> > Does statistic
> >     based optimization work for them?
>
> Yes.
>
> > Why is there no --explain function for them?
>
> The WHERE part is just a query so extract it, add "SELECT *" and use the
> query tools.
>
> > Would CONSTRUCTs be faster?
> No.
>
> >     2. optimization methods (fixed vs. stats) - which is a default one?
> Fixed.
>
> > how
> >     to run tdbstats against running Fuseki server?
> I'm afraid you can't.
>
> > how to run tdbstats on a
> >     database with triples in multiple named graphs (I have to use
> separate
> >     named graphs and I can't use default union graph)
>
> Are you sure it is the BGP part that is slow?
>
>      Andy
>
> >
> >
> > Thanks,
> > Piotr
> >
> >
> > [1]
> >
> https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
> >
>

Re: Jena performance optimization

Posted by Andy Seaborne <an...@apache.org>.

On 10/09/2020 12:16, Piotr Nowara wrote:
> Dear Community,
> 
> could you point me to resources on Jena performance optimization?
> 
> I found this excellent presentation [1] covering a lot of interesting
> topics but I'm not sure how up-to-date it might be. I also read the Jena
> docs which gave me some hints but also created some confusion. My main
> areas of interest are:
> 
>     1. How to make complex INSERT WHERE statements faster?

Usually, by reordering/rewriting the WHERE clause based on knowing the 
data better than the system.

The stats optimizer is only for BGPs anyway - the transformation of the 
alegbra is done separately.

> Does statistic
>     based optimization work for them? 

Yes.

> Why is there no --explain function for them?

The WHERE part is just a query so extract it, add "SELECT *" and use the 
query tools.

> Would CONSTRUCTs be faster?
No.

>     2. optimization methods (fixed vs. stats) - which is a default one?
Fixed.

> how
>     to run tdbstats against running Fuseki server?
I'm afraid you can't.

> how to run tdbstats on a
>     database with triples in multiple named graphs (I have to use separate
>     named graphs and I can't use default union graph)

Are you sure it is the BGP part that is slow?

     Andy

> 
> 
> Thanks,
> Piotr
> 
> 
> [1]
> https://events.static.linuxfound.org/sites/events/files/slides/SPARQL%20Optimisation%20101%20Tutorial.pdf
>