You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Mikael Pesonen <mi...@lingsoft.fi> on 2024/11/14 11:47:54 UTC
Re: SPAM-LOW: Slow clear graph
Thanks for the explanation! Yes it's mainly happening on disk, CPU usage
is at 0.5%. Are there any strategies to get around this? Somehow
compartmentalize the data so that deleting one compartment would be more
efficient?
On 14/11/2024 13:30, Rob @ DNR wrote:
> Details matter here e.g. what storage layer is in use? How big is the graph being deleted? How many other graphs (and triples) are in the server as a whole? You say a curl request so can we assume Fuseki? Are there other secondary indices involved e.g. Jena Text?
>
> ---
>
> Most Jena storage, i.e. TDB/TDB2, is quad-oriented behind the scenes so when you issue a CLEAR GRAPH <uri> (or a DROP GRAPH <uri>) what happens internally is that it must scan each index and delete all quads with the relevant <uri> in the graph position of the quad. For indexes where graph is later in the order e.g. SPOG these quads could be scattered across the entire index affecting many blocks on disk meaning the whole index needs to be read.
>
> For TDB2 which uses copy on write data structures this might also end up effectively having to rewrite every single block in the index which for large datasets could take an exceedingly long time.
>
> If you have secondary indices involved, e.g. Jena Text, then it is also potentially having to make the relevant delete requests to those indices as well.
>
> ---
>
> So, my guess would be that you have a lot of disk IO happening on your server if you happened to look at its resource consumption while the CLEAR GRAPH is ongoing?
>
> Rob
>
>
> From: Mikael Pesonen <mi...@lingsoft.fi>
> Date: Thursday, 14 November 2024 at 09:21
> To: users@jena.apache.org <us...@jena.apache.org>
> Subject: SPAM-LOW: Slow clear graph
> Curl command is running now over 24 hours with Jena, what could cause
> that? Shouldn't clear graph always be done in few seconds? It's not an
> expensive operation?
>
--
Lingsoft - 30 years of Leading Language Management
www.lingsoft.fi
Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books
Mikael Pesonen
Semantic Technologies
e-mail: mikael.pesonen@lingsoft.fi
Tel. +358 2 279 3300
Time zone: GMT+2
Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND
Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND
Re: Slow clear graph
Posted by Andy Seaborne <an...@apache.org>.
On 20/11/2024 11:38, Mikael Pesonen wrote:
> After 4 days I had to stop the query and it causes broken db, resulting
> error "NodeTableTRDF/Read" to following queries. This has happened many
> times before..
If there is something reproducible please provide it. However, if it is
environmental, such as other processes access the database files, then
there isn't much that Jena can do.
Andy
> On 19/11/2024 11:36, Mikael Pesonen wrote:
>> I guess best way to go now for our case is to use separate instances
>> of Jena and use federated queries? Downside is that more RAM is needed.
>>
>> On 18/11/2024 16:06, Andy Seaborne wrote:
>>>
>>> CLEAR and DROP are done inside a transaction. The SPARQL Update
>>> implementation is skewed to many, small operations.
>>>
>>> Looking at TDB2, that isn't just updating G??? indexes To make union
>>> default graph work, there are also indexes ???G. It's write
>>> amplification.
>>>
>>> There is a tradeoff - efficient union default graph vs easy of deletion.
>>>
>>> A different design could have each graph separately, optimizing for
>>> whole graph operations.
>>>
>>> A way round this, in the future, might be to employ server reload
>>> (WIP). Prepare a new dataset offline and do a zero-downtime reload of
>>> the configuration.
>>>
>>> Andy
>>>
>>> On 14/11/2024 11:47, Mikael Pesonen wrote:
>>>> Thanks for the explanation! Yes it's mainly happening on disk, CPU
>>>> usage is at 0.5%. Are there any strategies to get around this?
>>>> Somehow compartmentalize the data so that deleting one compartment
>>>> would be more efficient?
>>>>
>>>> On 14/11/2024 13:30, Rob @ DNR wrote:
>>>>> Details matter here e.g. what storage layer is in use? How big is
>>>>> the graph being deleted? How many other graphs (and triples) are
>>>>> in the server as a whole? You say a curl request so can we assume
>>>>> Fuseki? Are there other secondary indices involved e.g. Jena Text?
>>>>>
>>>>> ---
>>>>>
>>>>> Most Jena storage, i.e. TDB/TDB2, is quad-oriented behind the
>>>>> scenes so when you issue a CLEAR GRAPH <uri> (or a DROP GRAPH
>>>>> <uri>) what happens internally is that it must scan each index and
>>>>> delete all quads with the relevant <uri> in the graph position of
>>>>> the quad. For indexes where graph is later in the order e.g. SPOG
>>>>> these quads could be scattered across the entire index affecting
>>>>> many blocks on disk meaning the whole index needs to be read.
>>>>>
>>>>> For TDB2 which uses copy on write data structures this might also
>>>>> end up effectively having to rewrite every single block in the
>>>>> index which for large datasets could take an exceedingly long time.
>>>>>
>>>>> If you have secondary indices involved, e.g. Jena Text, then it is
>>>>> also potentially having to make the relevant delete requests to
>>>>> those indices as well.
>>>>>
>>>>> ---
>>>>>
>>>>> So, my guess would be that you have a lot of disk IO happening on
>>>>> your server if you happened to look at its resource consumption
>>>>> while the CLEAR GRAPH is ongoing?
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>> From: Mikael Pesonen <mi...@lingsoft.fi>
>>>>> Date: Thursday, 14 November 2024 at 09:21
>>>>> To: users@jena.apache.org <us...@jena.apache.org>
>>>>> Subject: SPAM-LOW: Slow clear graph
>>>>> Curl command is running now over 24 hours with Jena, what could cause
>>>>> that? Shouldn't clear graph always be done in few seconds? It's not an
>>>>> expensive operation?
>>>>>
>>>
Re: Slow clear graph
Posted by Mikael Pesonen <mi...@lingsoft.fi>.
After 4 days I had to stop the query and it causes broken db, resulting
error "NodeTableTRDF/Read" to following queries. This has happened many
times before..
On 19/11/2024 11:36, Mikael Pesonen wrote:
> I guess best way to go now for our case is to use separate instances
> of Jena and use federated queries? Downside is that more RAM is needed.
>
> On 18/11/2024 16:06, Andy Seaborne wrote:
>>
>> CLEAR and DROP are done inside a transaction. The SPARQL Update
>> implementation is skewed to many, small operations.
>>
>> Looking at TDB2, that isn't just updating G??? indexes To make union
>> default graph work, there are also indexes ???G. It's write
>> amplification.
>>
>> There is a tradeoff - efficient union default graph vs easy of deletion.
>>
>> A different design could have each graph separately, optimizing for
>> whole graph operations.
>>
>> A way round this, in the future, might be to employ server reload
>> (WIP). Prepare a new dataset offline and do a zero-downtime reload of
>> the configuration.
>>
>> Andy
>>
>> On 14/11/2024 11:47, Mikael Pesonen wrote:
>>> Thanks for the explanation! Yes it's mainly happening on disk, CPU
>>> usage is at 0.5%. Are there any strategies to get around this?
>>> Somehow compartmentalize the data so that deleting one compartment
>>> would be more efficient?
>>>
>>> On 14/11/2024 13:30, Rob @ DNR wrote:
>>>> Details matter here e.g. what storage layer is in use? How big is
>>>> the graph being deleted? How many other graphs (and triples) are
>>>> in the server as a whole? You say a curl request so can we assume
>>>> Fuseki? Are there other secondary indices involved e.g. Jena Text?
>>>>
>>>> ---
>>>>
>>>> Most Jena storage, i.e. TDB/TDB2, is quad-oriented behind the
>>>> scenes so when you issue a CLEAR GRAPH <uri> (or a DROP GRAPH
>>>> <uri>) what happens internally is that it must scan each index and
>>>> delete all quads with the relevant <uri> in the graph position of
>>>> the quad. For indexes where graph is later in the order e.g. SPOG
>>>> these quads could be scattered across the entire index affecting
>>>> many blocks on disk meaning the whole index needs to be read.
>>>>
>>>> For TDB2 which uses copy on write data structures this might also
>>>> end up effectively having to rewrite every single block in the
>>>> index which for large datasets could take an exceedingly long time.
>>>>
>>>> If you have secondary indices involved, e.g. Jena Text, then it is
>>>> also potentially having to make the relevant delete requests to
>>>> those indices as well.
>>>>
>>>> ---
>>>>
>>>> So, my guess would be that you have a lot of disk IO happening on
>>>> your server if you happened to look at its resource consumption
>>>> while the CLEAR GRAPH is ongoing?
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Mikael Pesonen <mi...@lingsoft.fi>
>>>> Date: Thursday, 14 November 2024 at 09:21
>>>> To: users@jena.apache.org <us...@jena.apache.org>
>>>> Subject: SPAM-LOW: Slow clear graph
>>>> Curl command is running now over 24 hours with Jena, what could cause
>>>> that? Shouldn't clear graph always be done in few seconds? It's not an
>>>> expensive operation?
>>>>
>>
Re: SPAM-LOW: Slow clear graph
Posted by Mikael Pesonen <mi...@lingsoft.fi>.
I guess best way to go now for our case is to use separate instances of
Jena and use federated queries? Downside is that more RAM is needed.
On 18/11/2024 16:06, Andy Seaborne wrote:
>
> CLEAR and DROP are done inside a transaction. The SPARQL Update
> implementation is skewed to many, small operations.
>
> Looking at TDB2, that isn't just updating G??? indexes To make union
> default graph work, there are also indexes ???G. It's write
> amplification.
>
> There is a tradeoff - efficient union default graph vs easy of deletion.
>
> A different design could have each graph separately, optimizing for
> whole graph operations.
>
> A way round this, in the future, might be to employ server reload
> (WIP). Prepare a new dataset offline and do a zero-downtime reload of
> the configuration.
>
> Andy
>
> On 14/11/2024 11:47, Mikael Pesonen wrote:
>> Thanks for the explanation! Yes it's mainly happening on disk, CPU
>> usage is at 0.5%. Are there any strategies to get around this?
>> Somehow compartmentalize the data so that deleting one compartment
>> would be more efficient?
>>
>> On 14/11/2024 13:30, Rob @ DNR wrote:
>>> Details matter here e.g. what storage layer is in use? How big is
>>> the graph being deleted? How many other graphs (and triples) are in
>>> the server as a whole? You say a curl request so can we assume
>>> Fuseki? Are there other secondary indices involved e.g. Jena Text?
>>>
>>> ---
>>>
>>> Most Jena storage, i.e. TDB/TDB2, is quad-oriented behind the scenes
>>> so when you issue a CLEAR GRAPH <uri> (or a DROP GRAPH <uri>) what
>>> happens internally is that it must scan each index and delete all
>>> quads with the relevant <uri> in the graph position of the quad.
>>> For indexes where graph is later in the order e.g. SPOG these quads
>>> could be scattered across the entire index affecting many blocks on
>>> disk meaning the whole index needs to be read.
>>>
>>> For TDB2 which uses copy on write data structures this might also
>>> end up effectively having to rewrite every single block in the index
>>> which for large datasets could take an exceedingly long time.
>>>
>>> If you have secondary indices involved, e.g. Jena Text, then it is
>>> also potentially having to make the relevant delete requests to
>>> those indices as well.
>>>
>>> ---
>>>
>>> So, my guess would be that you have a lot of disk IO happening on
>>> your server if you happened to look at its resource consumption
>>> while the CLEAR GRAPH is ongoing?
>>>
>>> Rob
>>>
>>>
>>> From: Mikael Pesonen <mi...@lingsoft.fi>
>>> Date: Thursday, 14 November 2024 at 09:21
>>> To: users@jena.apache.org <us...@jena.apache.org>
>>> Subject: SPAM-LOW: Slow clear graph
>>> Curl command is running now over 24 hours with Jena, what could cause
>>> that? Shouldn't clear graph always be done in few seconds? It's not an
>>> expensive operation?
>>>
>
--
Lingsoft - 30 years of Leading Language Management
www.lingsoft.fi
Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books
Mikael Pesonen
Semantic Technologies
e-mail: mikael.pesonen@lingsoft.fi
Tel. +358 2 279 3300
Time zone: GMT+2
Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND
Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND
Re: SPAM-LOW: Slow clear graph
Posted by Andy Seaborne <an...@apache.org>.
CLEAR and DROP are done inside a transaction. The SPARQL Update
implementation is skewed to many, small operations.
Looking at TDB2, that isn't just updating G??? indexes To make union
default graph work, there are also indexes ???G. It's write amplification.
There is a tradeoff - efficient union default graph vs easy of deletion.
A different design could have each graph separately, optimizing for
whole graph operations.
A way round this, in the future, might be to employ server reload (WIP).
Prepare a new dataset offline and do a zero-downtime reload of the
configuration.
Andy
On 14/11/2024 11:47, Mikael Pesonen wrote:
> Thanks for the explanation! Yes it's mainly happening on disk, CPU usage
> is at 0.5%. Are there any strategies to get around this? Somehow
> compartmentalize the data so that deleting one compartment would be more
> efficient?
>
> On 14/11/2024 13:30, Rob @ DNR wrote:
>> Details matter here e.g. what storage layer is in use? How big is the
>> graph being deleted? How many other graphs (and triples) are in the
>> server as a whole? You say a curl request so can we assume Fuseki?
>> Are there other secondary indices involved e.g. Jena Text?
>>
>> ---
>>
>> Most Jena storage, i.e. TDB/TDB2, is quad-oriented behind the scenes
>> so when you issue a CLEAR GRAPH <uri> (or a DROP GRAPH <uri>) what
>> happens internally is that it must scan each index and delete all
>> quads with the relevant <uri> in the graph position of the quad. For
>> indexes where graph is later in the order e.g. SPOG these quads could
>> be scattered across the entire index affecting many blocks on disk
>> meaning the whole index needs to be read.
>>
>> For TDB2 which uses copy on write data structures this might also end
>> up effectively having to rewrite every single block in the index which
>> for large datasets could take an exceedingly long time.
>>
>> If you have secondary indices involved, e.g. Jena Text, then it is
>> also potentially having to make the relevant delete requests to those
>> indices as well.
>>
>> ---
>>
>> So, my guess would be that you have a lot of disk IO happening on your
>> server if you happened to look at its resource consumption while the
>> CLEAR GRAPH is ongoing?
>>
>> Rob
>>
>>
>> From: Mikael Pesonen <mi...@lingsoft.fi>
>> Date: Thursday, 14 November 2024 at 09:21
>> To: users@jena.apache.org <us...@jena.apache.org>
>> Subject: SPAM-LOW: Slow clear graph
>> Curl command is running now over 24 hours with Jena, what could cause
>> that? Shouldn't clear graph always be done in few seconds? It's not an
>> expensive operation?
>>