You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Balduin Landolt <ba...@dasch.swiss.INVALID> on 2024/04/22 15:22:56 UTC
Fuseki growing in size and need for compaction
Hello,
we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
essentially the same) with roughly 40 Mio triples (tendency: growing).
Not sure what configuration is relevant, but we have the default graph as
the union graph.
Also, we use Fuseki as our main database, not just as a "view on our data"
so we do quite a bit of updating on the data all the time.
Lately, we've been having more and more issues with servers running out of
disk space because Fuseki's database grew pretty rapidly.
This can be solved by compacting the DB, but with our data and hardware
this takes ca. 15 minutes, during which Fuseki does not accept any update
queries, so for the production system we can't really do this outside of
nighttime hours when (hopefully) no one uses the system anyways.
Some things we've noticed:
- A subset of our data (I think ~20 Mio triples) taking up 6GB in compacted
state, when dumped to a .trig file is ca. 5GB. But when uploading the same
.trig file to an empty DB, this grows to ca. 25GB
- Dropping graphs does not free up disk space
- A sequence of e.g. 10k queries updating only a small number of triples
(maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
like 10s to 100s of GB (I don't have numbers on this one, but it was
substantial).
My question is:
Would that kind of growth in disk usage be expected? Are other people
having similar issues? Are there strategies to mitigate this? Maybe some
configuration that may be tweaked or so?
Best & thanks in advance,
Balduin
TDB3
Posted by Andy Seaborne <an...@apache.org>.
On 24/04/2024 21:42, Martynas Jusevičius wrote:
> Andy,
>
> Not directly related, but would different storage backend address
> issues like this?
>
> It might sound a bit like the legacy SDB, but AFAIK oxigraph, Stardog
> and another commercial triplestore use RocksDB for storage.
> https://github.com/oxigraph/oxigraph
> https://docs.stardog.com/operating-stardog/database-administration/storage-optimize
>
> There is even a RocksDB backend for Jena:
> https://github.com/zourzouvillys/triplerocks
> And just now I found your own TDB3 repo: https://github.com/afs/TDB3
>
> Can you shed some light on TDB3 and this approach in general?
TDB3 uses RocksDB as the storage layer replacing the custom B+trees and
also the node table. It's a naive use of RockDB. It seems to work
(functional). It's untested both in code and deployment.
It loads slower than tdb2 bulk loaders (IIRC maybe 70Ktriples/s) but
little work has been done to exploit Rocks capabilities.
The advantage of Rocks is that it is likely to be around for a long time
(= it's a safe investment), it's transactional, has compression [1], has
compaction [2], it has a java wrapper (separate, but closely related and
in contact with the Rocks team).
While there are many storage engines that claim to be faster than
RocksDB, often, such claims have assumptions.
There are other storage layers to explore as well.
Andy
[1] Better, or also, would probably be compression in the encoding of
stored tuples.
[2] compaction has two parts - finding the RDF terms that are currently
in use in the database and recovering space indexes. RocksDB compaction
is about the second case.
>
> Martynas
>
> On Wed, Apr 24, 2024 at 10:30 PM Andy Seaborne <an...@apache.org> wrote:
>>
>> Hi Balduin,
>>
>> Thanks for the detailed report. It's useful to hear of the use case that
>> occur and also the behaviour of specific deployments.
>>
>> On 22/04/2024 16:22, Balduin Landolt wrote:
>>> Hello,
>>>
>>> we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
>>> essentially the same) with roughly 40 Mio triples (tendency: growing).
>>> Not sure what configuration is relevant, but we have the default graph as
>>> the union graph.
>>
>> Sort of relevant.
>>
>> There are more indexes on named graphs so there is more compaction work
>> to be done.
>>
>> "union default graph" is a view at query time, not in the storage itself.
>>
>>> Also, we use Fuseki as our main database, not just as a "view on our data"
>>> so we do quite a bit of updating on the data all the time.
>>>
>>> Lately, we've been having more and more issues with servers running out of
>>> disk space because Fuseki's database grew pretty rapidly.
>>> This can be solved by compacting the DB, but with our data and hardware
>>> this takes ca. 15 minutes, during which Fuseki does not accept any update
>>> queries, so for the production system we can't really do this outside of
>>> nighttime hours when (hopefully) no one uses the system anyways.
>>
>> Is the database disk area on an SSD, on a hard disk, or a remote
>> filesystem (and then, is it SSD or hard disk)?
>>
>>> Some things we've noticed:
>>> - A subset of our data (I think ~20 Mio triples) taking up 6GB in compacted
>>> state, when dumped to a .trig file is ca. 5GB. But when uploading the same
>>> .trig file to an empty DB, this grows to ca. 25GB
>>> - Dropping graphs does not free up disk space
>>
>> That's at the point the graph is dropped? It should reclaim space at
>> compaction.
>>
>>> - A sequence of e.g. 10k queries updating only a small number of triples
>>> (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
>>> like 10s to 100s of GB (I don't have numbers on this one, but it was
>>> substantial).
>>
>> This might be a factor. There is a space overhead per transaction, not
>> solely due to the size of update. Sounds like 10k updates is makiing
>> that appreciably.
>>
>> Are the updates all additions? Or a mix of additions and deletions?
>>
>>> My question is:
>>
>>> Would that kind of growth in disk usage be expected?
>>
>> Given 10K updates, then what you describe sounds possible.
>>
>> > Are other people having similar issues?> Are there strategies to
>> mitigate this?
>> Batching the updates although this does mean the updates don't
>> immediately appear in the database.
>>
>> This can work reasonable when the updates are additions. If there are
>> deletes, it's harder.
>>
>>> Maybe some configuration that may be tweaked or so?
>>
>> Sorry - there aren't any controls.
>>
>>>
>>> Best & thanks in advance,
>>> Balduin
>>>
>>
>> Andy
Re: Fuseki growing in size and need for compaction
Posted by Martynas Jusevičius <ma...@atomgraph.com>.
Andy,
Not directly related, but would different storage backend address
issues like this?
It might sound a bit like the legacy SDB, but AFAIK oxigraph, Stardog
and another commercial triplestore use RocksDB for storage.
https://github.com/oxigraph/oxigraph
https://docs.stardog.com/operating-stardog/database-administration/storage-optimize
There is even a RocksDB backend for Jena:
https://github.com/zourzouvillys/triplerocks
And just now I found your own TDB3 repo: https://github.com/afs/TDB3
Can you shed some light on TDB3 and this approach in general?
Martynas
On Wed, Apr 24, 2024 at 10:30 PM Andy Seaborne <an...@apache.org> wrote:
>
> Hi Balduin,
>
> Thanks for the detailed report. It's useful to hear of the use case that
> occur and also the behaviour of specific deployments.
>
> On 22/04/2024 16:22, Balduin Landolt wrote:
> > Hello,
> >
> > we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
> > essentially the same) with roughly 40 Mio triples (tendency: growing).
> > Not sure what configuration is relevant, but we have the default graph as
> > the union graph.
>
> Sort of relevant.
>
> There are more indexes on named graphs so there is more compaction work
> to be done.
>
> "union default graph" is a view at query time, not in the storage itself.
>
> > Also, we use Fuseki as our main database, not just as a "view on our data"
> > so we do quite a bit of updating on the data all the time.
> >
> > Lately, we've been having more and more issues with servers running out of
> > disk space because Fuseki's database grew pretty rapidly.
> > This can be solved by compacting the DB, but with our data and hardware
> > this takes ca. 15 minutes, during which Fuseki does not accept any update
> > queries, so for the production system we can't really do this outside of
> > nighttime hours when (hopefully) no one uses the system anyways.
>
> Is the database disk area on an SSD, on a hard disk, or a remote
> filesystem (and then, is it SSD or hard disk)?
>
> > Some things we've noticed:
> > - A subset of our data (I think ~20 Mio triples) taking up 6GB in compacted
> > state, when dumped to a .trig file is ca. 5GB. But when uploading the same
> > .trig file to an empty DB, this grows to ca. 25GB
> > - Dropping graphs does not free up disk space
>
> That's at the point the graph is dropped? It should reclaim space at
> compaction.
>
> > - A sequence of e.g. 10k queries updating only a small number of triples
> > (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
> > like 10s to 100s of GB (I don't have numbers on this one, but it was
> > substantial).
>
> This might be a factor. There is a space overhead per transaction, not
> solely due to the size of update. Sounds like 10k updates is makiing
> that appreciably.
>
> Are the updates all additions? Or a mix of additions and deletions?
>
> > My question is:
>
> > Would that kind of growth in disk usage be expected?
>
> Given 10K updates, then what you describe sounds possible.
>
> > Are other people having similar issues?> Are there strategies to
> mitigate this?
> Batching the updates although this does mean the updates don't
> immediately appear in the database.
>
> This can work reasonable when the updates are additions. If there are
> deletes, it's harder.
>
> > Maybe some configuration that may be tweaked or so?
>
> Sorry - there aren't any controls.
>
> >
> > Best & thanks in advance,
> > Balduin
> >
>
> Andy
Re: Fuseki growing in size and need for compaction
Posted by Andy Seaborne <an...@apache.org>.
Hi Balduin,
Thanks for the detailed report. It's useful to hear of the use case that
occur and also the behaviour of specific deployments.
On 22/04/2024 16:22, Balduin Landolt wrote:
> Hello,
>
> we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
> essentially the same) with roughly 40 Mio triples (tendency: growing).
> Not sure what configuration is relevant, but we have the default graph as
> the union graph.
Sort of relevant.
There are more indexes on named graphs so there is more compaction work
to be done.
"union default graph" is a view at query time, not in the storage itself.
> Also, we use Fuseki as our main database, not just as a "view on our data"
> so we do quite a bit of updating on the data all the time.
>
> Lately, we've been having more and more issues with servers running out of
> disk space because Fuseki's database grew pretty rapidly.
> This can be solved by compacting the DB, but with our data and hardware
> this takes ca. 15 minutes, during which Fuseki does not accept any update
> queries, so for the production system we can't really do this outside of
> nighttime hours when (hopefully) no one uses the system anyways.
Is the database disk area on an SSD, on a hard disk, or a remote
filesystem (and then, is it SSD or hard disk)?
> Some things we've noticed:
> - A subset of our data (I think ~20 Mio triples) taking up 6GB in compacted
> state, when dumped to a .trig file is ca. 5GB. But when uploading the same
> .trig file to an empty DB, this grows to ca. 25GB
> - Dropping graphs does not free up disk space
That's at the point the graph is dropped? It should reclaim space at
compaction.
> - A sequence of e.g. 10k queries updating only a small number of triples
> (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
> like 10s to 100s of GB (I don't have numbers on this one, but it was
> substantial).
This might be a factor. There is a space overhead per transaction, not
solely due to the size of update. Sounds like 10k updates is makiing
that appreciably.
Are the updates all additions? Or a mix of additions and deletions?
> My question is:
> Would that kind of growth in disk usage be expected?
Given 10K updates, then what you describe sounds possible.
> Are other people having similar issues?> Are there strategies to
mitigate this?
Batching the updates although this does mean the updates don't
immediately appear in the database.
This can work reasonable when the updates are additions. If there are
deletes, it's harder.
> Maybe some configuration that may be tweaked or so?
Sorry - there aren't any controls.
>
> Best & thanks in advance,
> Balduin
>
Andy
Re: Fuseki growing in size and need for compaction
Posted by Balduin Landolt <ba...@dasch.swiss.INVALID>.
Hi Milorad,
I'm afraid I don't have a very satisfying response to give you. We never
found a way to solve the problem, just ways to work around it:
- We have a DB compaction scheduled nightly, during the time we have fewest
users.
- We make sure to have enough disk space available to last until the next
scheduled compaction.
- If we have an internal process that we know will make a large number of
updates, we make sure to batch it, so that only an acceptable number of
updates is performed before it suspends until the next compaction has
happened.
- We have our data logically compartmentalized in named graphs; if there is
a process that will perform many updates, wherever possible we design it in
such a way that it will only affect a single graph. Like this we can run it
on a separate instance with a mirror of our production database, and once
the process has finished, we simply upload the graph in question from the
other instance to prod. (Though this is admittedly dangerous in terms of
data integrity.)
For the longer term, we have considered and discarded two solutions: 1)
swapping Fuseki out for a different triple store, or 2) changing our
application so that the triplestore no longer is the source of truth, but
instead just a secondary DB for SPARQL querying. Both would be substantial
changes and each come with some cost and complication.
Instead, what we're now doing is splitting out some functionality of our
current (monolithic) system, into separate components that will no longer
rely on Fuseki, hoping that this will reduce the number of updates to an
extent where the disk size won't be such an issue anymore.
Best,
Balduin
On Thu, Dec 26, 2024 at 3:48 PM Milorad Tosic <mb...@yahoo.com.invalid>
wrote:
> Hi Balduin,
>
> We have a similar problem. Could you let me know status on your progress?
>
> Thanks
>
> Milorad
>
> On 4/22/2024 5:22 PM, Balduin Landolt wrote:
> > Hello,
> >
> > we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
> > essentially the same) with roughly 40 Mio triples (tendency: growing).
> > Not sure what configuration is relevant, but we have the default graph as
> > the union graph.
> > Also, we use Fuseki as our main database, not just as a "view on our
> data"
> > so we do quite a bit of updating on the data all the time.
> >
> > Lately, we've been having more and more issues with servers running out
> of
> > disk space because Fuseki's database grew pretty rapidly.
> > This can be solved by compacting the DB, but with our data and hardware
> > this takes ca. 15 minutes, during which Fuseki does not accept any update
> > queries, so for the production system we can't really do this outside of
> > nighttime hours when (hopefully) no one uses the system anyways.
> >
> > Some things we've noticed:
> > - A subset of our data (I think ~20 Mio triples) taking up 6GB in
> compacted
> > state, when dumped to a .trig file is ca. 5GB. But when uploading the
> same
> > .trig file to an empty DB, this grows to ca. 25GB
> > - Dropping graphs does not free up disk space
> > - A sequence of e.g. 10k queries updating only a small number of triples
> > (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
> > like 10s to 100s of GB (I don't have numbers on this one, but it was
> > substantial).
> >
> > My question is:
> > Would that kind of growth in disk usage be expected? Are other people
> > having similar issues? Are there strategies to mitigate this? Maybe some
> > configuration that may be tweaked or so?
> >
> > Best & thanks in advance,
> > Balduin
> >
>
Re: Fuseki growing in size and need for compaction
Posted by Andy Seaborne <an...@apache.org>.
On 26/12/2024 14:47, Milorad Tosic wrote:
> Hi Balduin,
>
> We have a similar problem. Could you let me know status on your progress?
>
> Thanks
>
> Milorad
Hi Milorad,
Could you provide some background about your usage of Jena?
How much data does the store have?
Was it started from empty or was there an initial bulk load?
What is the pattern of usage?
Frequency of queries?
Frequency of updates and size of updates? Do you delete or replace data
or only add more data?
Andy
>
> On 4/22/2024 5:22 PM, Balduin Landolt wrote:
>> Hello,
>>
>> we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
>> essentially the same) with roughly 40 Mio triples (tendency: growing).
>> Not sure what configuration is relevant, but we have the default graph as
>> the union graph.
>> Also, we use Fuseki as our main database, not just as a "view on our
>> data"
>> so we do quite a bit of updating on the data all the time.
>>
>> Lately, we've been having more and more issues with servers running
>> out of
>> disk space because Fuseki's database grew pretty rapidly.
>> This can be solved by compacting the DB, but with our data and hardware
>> this takes ca. 15 minutes, during which Fuseki does not accept any update
>> queries, so for the production system we can't really do this outside of
>> nighttime hours when (hopefully) no one uses the system anyways.
>>
>> Some things we've noticed:
>> - A subset of our data (I think ~20 Mio triples) taking up 6GB in
>> compacted
>> state, when dumped to a .trig file is ca. 5GB. But when uploading the
>> same
>> .trig file to an empty DB, this grows to ca. 25GB
>> - Dropping graphs does not free up disk space
>> - A sequence of e.g. 10k queries updating only a small number of triples
>> (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
>> like 10s to 100s of GB (I don't have numbers on this one, but it was
>> substantial).
>>
>> My question is:
>> Would that kind of growth in disk usage be expected? Are other people
>> having similar issues? Are there strategies to mitigate this? Maybe some
>> configuration that may be tweaked or so?
>>
>> Best & thanks in advance,
>> Balduin
>>
Re: Fuseki growing in size and need for compaction
Posted by Milorad Tosic <mb...@yahoo.com.INVALID>.
Hi Balduin,
We have a similar problem. Could you let me know status on your progress?
Thanks
Milorad
On 4/22/2024 5:22 PM, Balduin Landolt wrote:
> Hello,
>
> we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
> essentially the same) with roughly 40 Mio triples (tendency: growing).
> Not sure what configuration is relevant, but we have the default graph as
> the union graph.
> Also, we use Fuseki as our main database, not just as a "view on our data"
> so we do quite a bit of updating on the data all the time.
>
> Lately, we've been having more and more issues with servers running out of
> disk space because Fuseki's database grew pretty rapidly.
> This can be solved by compacting the DB, but with our data and hardware
> this takes ca. 15 minutes, during which Fuseki does not accept any update
> queries, so for the production system we can't really do this outside of
> nighttime hours when (hopefully) no one uses the system anyways.
>
> Some things we've noticed:
> - A subset of our data (I think ~20 Mio triples) taking up 6GB in compacted
> state, when dumped to a .trig file is ca. 5GB. But when uploading the same
> .trig file to an empty DB, this grows to ca. 25GB
> - Dropping graphs does not free up disk space
> - A sequence of e.g. 10k queries updating only a small number of triples
> (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
> like 10s to 100s of GB (I don't have numbers on this one, but it was
> substantial).
>
> My question is:
> Would that kind of growth in disk usage be expected? Are other people
> having similar issues? Are there strategies to mitigate this? Maybe some
> configuration that may be tweaked or so?
>
> Best & thanks in advance,
> Balduin
>
Re: Fuseki growing in size and need for compaction
Posted by Mikael Pesonen <mi...@lingsoft.fi>.
We have the same issue and have stopped using Jena for that reason.
On 22/04/2024 18:22, Balduin Landolt wrote:
> Hello,
>
> we're running Fuseki 5.0.0 (but previously the last 4.x versions behaved
> essentially the same) with roughly 40 Mio triples (tendency: growing).
> Not sure what configuration is relevant, but we have the default graph as
> the union graph.
> Also, we use Fuseki as our main database, not just as a "view on our data"
> so we do quite a bit of updating on the data all the time.
>
> Lately, we've been having more and more issues with servers running out of
> disk space because Fuseki's database grew pretty rapidly.
> This can be solved by compacting the DB, but with our data and hardware
> this takes ca. 15 minutes, during which Fuseki does not accept any update
> queries, so for the production system we can't really do this outside of
> nighttime hours when (hopefully) no one uses the system anyways.
>
> Some things we've noticed:
> - A subset of our data (I think ~20 Mio triples) taking up 6GB in compacted
> state, when dumped to a .trig file is ca. 5GB. But when uploading the same
> .trig file to an empty DB, this grows to ca. 25GB
> - Dropping graphs does not free up disk space
> - A sequence of e.g. 10k queries updating only a small number of triples
> (maybe 1-10 or so) on the full dataset seems to grow the DB size a lot,
> like 10s to 100s of GB (I don't have numbers on this one, but it was
> substantial).
>
> My question is:
> Would that kind of growth in disk usage be expected? Are other people
> having similar issues? Are there strategies to mitigate this? Maybe some
> configuration that may be tweaked or so?
>
> Best & thanks in advance,
> Balduin
>
--
Lingsoft - 30 years of Leading Language Management
www.lingsoft.fi
Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books
Mikael Pesonen
Semantic Technologies
e-mail: mikael.pesonen@lingsoft.fi
Tel. +358 2 279 3300
Time zone: GMT+2
Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND
Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND