You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by "Janda, Radim" <ra...@reporters.cz> on 2017/01/06 11:50:59 UTC
Fuseki - how to release memory
Hello,
We use Jena Fuseki to process a lot of small data sets.
It works in the following way:
1. Delete whole Fuseki (using DELETE command)
2. Load data to Fuseki (using INSERT)
3. Tranform data and create output (sparql called from Python)
4. ad 1)2)3 .... delete Fuseki and Transform another data set
We have found out that memory is not released after delete in Fuseki.
That means we have lack of memory after some data sets are transformed.
Actually we restart Fuseki server after some number of data sets but we
are looking for the better solution.
Can you please help us with memory releasing?
Many thanks
Radim
Re: Fuseki - how to release memory
Posted by "A. Soroka" <aj...@virginia.edu>.
Can you give us your actual Fuseki config (i.e. assembler file)? Or are you repeatedly creating new datasets via the admin API?
---
A. Soroka
The University of Virginia Library
> On Jan 6, 2017, at 10:43 AM, Janda, Radim <ra...@reporters.cz> wrote:
>
> Hello,
> we use in-memory datasets.
> JVM is big enough but as we process thousands of small data sets the memory
> is allocated continuously.
> Actualy we restart Fuseki every hour to avoid out of memory error.
> However the performance is also decreasing in time (before restart) that's
> why we are looking for the possibility of memory cleanup.
>
> Radim
>
> On Fri, Jan 6, 2017 at 4:12 PM, Andy Seaborne <an...@apache.org> wrote:
>
>> Are you using persistent or an in-memory datasets for your working storage?
>>
>> If you really mean memory (RAM), are you sure the JVM is big enough?
>>
>> Fuseki tries to avoid holding on to cache transactions but if the server
>> is under heavy read requests (Rob's point) then it can build up (solution -
>> reduce the read load for a short while) - also TDB does try to switch to
>> emergency measures after a while but maybe before then the RAM usage has
>> grown too much.
>>
>> Andy
>>
>>
>> On 06/01/17 14:07, Rob Vesse wrote:
>>
>>> Deleting data does not reclaim all the memory, exactly what is and isn’t
>>> reclaimed depends somewhat on your exact usage pattern.
>>>
>>> The B+Tree’s which are the primary data structure for TDB, the default
>>> database used in Fuseki, does not reclaim the space. It is potentially
>>> subject fragmentation as well so memory used tends to grow over time. The
>>> node table portion of the database, the mapping from RDF terms to internal
>>> database identifiers is a sequential data structure that will only ever
>>> grow over time. It is also worth noting that many of the data structures
>>> are backed by memory mapped files which are off-heap and subject to the
>>> vagaries of how your OS handles this.
>>>
>>> Additionally, if you place Fuseki under continuous load TDB maybe blocked
>>> from writing the in memory journal back to disk which can cause back to
>>> grow unbounded overtime and prevent memory being reclaimed. Adding
>>> occasional pauses between operations can help to alleviate this.
>>>
>>> As Lorenz notes for this kind of use case you may not need Fuseki at all
>>> and could simply drive TDB programmatically instead.
>>>
>>> As a general point creating a fresh database rather than reusing an
>>> existing one will much more efficiently use memory. However, if you’re
>>> running on Windows then there is a known OS specific JVM bug that can cause
>>> memory mapped files to not be properly deleted until after the process
>>> exits.
>>>
>>> Rob
>>>
>>> On 06/01/2017 12:23, "Janda, Radim" <ra...@reporters.cz> wrote:
>>>
>>> Hello Lorenz,
>>> yes I meant delete data from Fuseki using DELETE command.
>>> We have version 2.4 installed.
>>> We use two types of queries:
>>> 1. Insert new triples based on existing triples rdf model (insert
>>> sparql)
>>> 2. Find some results in the data (select sparql)
>>>
>>> Thanks
>>>
>>> Radim
>>>
>>> On Fri, Jan 6, 2017 at 1:04 PM, Lorenz B. <
>>> buehmann@informatik.uni-leipzig.de> wrote:
>>>
>>>> Hello Radim,
>>>>
>>>> just to avoid confusion, with "Delete whole Fuseki" you mean the
>>> data
>>>> loaded into Fuseki, right?
>>>>
>>>> Which Fuseki version do you use?
>>>>
>>>> What kind of transformation do you do? I'm asking because I'm
>>> wondering
>>>> if it's necessary to use Fuseki.
>>>>
>>>>
>>>>
>>>> Cheers,
>>>> Lorenz
>>>>
>>>>> Hello,
>>>>> We use Jena Fuseki to process a lot of small data sets.
>>>>>
>>>>> It works in the following way:
>>>>> 1. Delete whole Fuseki (using DELETE command)
>>>>> 2. Load data to Fuseki (using INSERT)
>>>>> 3. Tranform data and create output (sparql called from Python)
>>>>> 4. ad 1)2)3 .... delete Fuseki and Transform another data set
>>>>>
>>>>> We have found out that memory is not released after delete in
>>> Fuseki.
>>>>> That means we have lack of memory after some data sets are
>>> transformed.
>>>>> Actually we restart Fuseki server after some number of data sets
>>> but we
>>>>> are looking for the better solution.
>>>>>
>>>>> Can you please help us with memory releasing?
>>>>>
>>>>> Many thanks
>>>>>
>>>>> Radim
>>>>>
>>>> --
>>>> Lorenz Bühmann
>>>> AKSW group, University of Leipzig
>>>> Group: http://aksw.org - semantic web research center
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
Re: Fuseki - how to release memory
Posted by "Janda, Radim" <ra...@reporters.cz>.
Hello,
we use in-memory datasets.
JVM is big enough but as we process thousands of small data sets the memory
is allocated continuously.
Actualy we restart Fuseki every hour to avoid out of memory error.
However the performance is also decreasing in time (before restart) that's
why we are looking for the possibility of memory cleanup.
Radim
On Fri, Jan 6, 2017 at 4:12 PM, Andy Seaborne <an...@apache.org> wrote:
> Are you using persistent or an in-memory datasets for your working storage?
>
> If you really mean memory (RAM), are you sure the JVM is big enough?
>
> Fuseki tries to avoid holding on to cache transactions but if the server
> is under heavy read requests (Rob's point) then it can build up (solution -
> reduce the read load for a short while) - also TDB does try to switch to
> emergency measures after a while but maybe before then the RAM usage has
> grown too much.
>
> Andy
>
>
> On 06/01/17 14:07, Rob Vesse wrote:
>
>> Deleting data does not reclaim all the memory, exactly what is and isn’t
>> reclaimed depends somewhat on your exact usage pattern.
>>
>> The B+Tree’s which are the primary data structure for TDB, the default
>> database used in Fuseki, does not reclaim the space. It is potentially
>> subject fragmentation as well so memory used tends to grow over time. The
>> node table portion of the database, the mapping from RDF terms to internal
>> database identifiers is a sequential data structure that will only ever
>> grow over time. It is also worth noting that many of the data structures
>> are backed by memory mapped files which are off-heap and subject to the
>> vagaries of how your OS handles this.
>>
>> Additionally, if you place Fuseki under continuous load TDB maybe blocked
>> from writing the in memory journal back to disk which can cause back to
>> grow unbounded overtime and prevent memory being reclaimed. Adding
>> occasional pauses between operations can help to alleviate this.
>>
>> As Lorenz notes for this kind of use case you may not need Fuseki at all
>> and could simply drive TDB programmatically instead.
>>
>> As a general point creating a fresh database rather than reusing an
>> existing one will much more efficiently use memory. However, if you’re
>> running on Windows then there is a known OS specific JVM bug that can cause
>> memory mapped files to not be properly deleted until after the process
>> exits.
>>
>> Rob
>>
>> On 06/01/2017 12:23, "Janda, Radim" <ra...@reporters.cz> wrote:
>>
>> Hello Lorenz,
>> yes I meant delete data from Fuseki using DELETE command.
>> We have version 2.4 installed.
>> We use two types of queries:
>> 1. Insert new triples based on existing triples rdf model (insert
>> sparql)
>> 2. Find some results in the data (select sparql)
>>
>> Thanks
>>
>> Radim
>>
>> On Fri, Jan 6, 2017 at 1:04 PM, Lorenz B. <
>> buehmann@informatik.uni-leipzig.de> wrote:
>>
>> > Hello Radim,
>> >
>> > just to avoid confusion, with "Delete whole Fuseki" you mean the
>> data
>> > loaded into Fuseki, right?
>> >
>> > Which Fuseki version do you use?
>> >
>> > What kind of transformation do you do? I'm asking because I'm
>> wondering
>> > if it's necessary to use Fuseki.
>> >
>> >
>> >
>> > Cheers,
>> > Lorenz
>> >
>> > > Hello,
>> > > We use Jena Fuseki to process a lot of small data sets.
>> > >
>> > > It works in the following way:
>> > > 1. Delete whole Fuseki (using DELETE command)
>> > > 2. Load data to Fuseki (using INSERT)
>> > > 3. Tranform data and create output (sparql called from Python)
>> > > 4. ad 1)2)3 .... delete Fuseki and Transform another data set
>> > >
>> > > We have found out that memory is not released after delete in
>> Fuseki.
>> > > That means we have lack of memory after some data sets are
>> transformed.
>> > > Actually we restart Fuseki server after some number of data sets
>> but we
>> > > are looking for the better solution.
>> > >
>> > > Can you please help us with memory releasing?
>> > >
>> > > Many thanks
>> > >
>> > > Radim
>> > >
>> > --
>> > Lorenz Bühmann
>> > AKSW group, University of Leipzig
>> > Group: http://aksw.org - semantic web research center
>> >
>> >
>>
>>
>>
>>
>>
>>
Re: Fuseki - how to release memory
Posted by Andy Seaborne <an...@apache.org>.
Are you using persistent or an in-memory datasets for your working storage?
If you really mean memory (RAM), are you sure the JVM is big enough?
Fuseki tries to avoid holding on to cache transactions but if the server
is under heavy read requests (Rob's point) then it can build up
(solution - reduce the read load for a short while) - also TDB does try
to switch to emergency measures after a while but maybe before then the
RAM usage has grown too much.
Andy
On 06/01/17 14:07, Rob Vesse wrote:
> Deleting data does not reclaim all the memory, exactly what is and isn\u2019t reclaimed depends somewhat on your exact usage pattern.
>
> The B+Tree\u2019s which are the primary data structure for TDB, the default database used in Fuseki, does not reclaim the space. It is potentially subject fragmentation as well so memory used tends to grow over time. The node table portion of the database, the mapping from RDF terms to internal database identifiers is a sequential data structure that will only ever grow over time. It is also worth noting that many of the data structures are backed by memory mapped files which are off-heap and subject to the vagaries of how your OS handles this.
>
> Additionally, if you place Fuseki under continuous load TDB maybe blocked from writing the in memory journal back to disk which can cause back to grow unbounded overtime and prevent memory being reclaimed. Adding occasional pauses between operations can help to alleviate this.
>
> As Lorenz notes for this kind of use case you may not need Fuseki at all and could simply drive TDB programmatically instead.
>
> As a general point creating a fresh database rather than reusing an existing one will much more efficiently use memory. However, if you\u2019re running on Windows then there is a known OS specific JVM bug that can cause memory mapped files to not be properly deleted until after the process exits.
>
> Rob
>
> On 06/01/2017 12:23, "Janda, Radim" <ra...@reporters.cz> wrote:
>
> Hello Lorenz,
> yes I meant delete data from Fuseki using DELETE command.
> We have version 2.4 installed.
> We use two types of queries:
> 1. Insert new triples based on existing triples rdf model (insert sparql)
> 2. Find some results in the data (select sparql)
>
> Thanks
>
> Radim
>
> On Fri, Jan 6, 2017 at 1:04 PM, Lorenz B. <
> buehmann@informatik.uni-leipzig.de> wrote:
>
> > Hello Radim,
> >
> > just to avoid confusion, with "Delete whole Fuseki" you mean the data
> > loaded into Fuseki, right?
> >
> > Which Fuseki version do you use?
> >
> > What kind of transformation do you do? I'm asking because I'm wondering
> > if it's necessary to use Fuseki.
> >
> >
> >
> > Cheers,
> > Lorenz
> >
> > > Hello,
> > > We use Jena Fuseki to process a lot of small data sets.
> > >
> > > It works in the following way:
> > > 1. Delete whole Fuseki (using DELETE command)
> > > 2. Load data to Fuseki (using INSERT)
> > > 3. Tranform data and create output (sparql called from Python)
> > > 4. ad 1)2)3 .... delete Fuseki and Transform another data set
> > >
> > > We have found out that memory is not released after delete in Fuseki.
> > > That means we have lack of memory after some data sets are transformed.
> > > Actually we restart Fuseki server after some number of data sets but we
> > > are looking for the better solution.
> > >
> > > Can you please help us with memory releasing?
> > >
> > > Many thanks
> > >
> > > Radim
> > >
> > --
> > Lorenz B�hmann
> > AKSW group, University of Leipzig
> > Group: http://aksw.org - semantic web research center
> >
> >
>
>
>
>
>
Re: Fuseki - how to release memory
Posted by Rob Vesse <rv...@dotnetrdf.org>.
Deleting data does not reclaim all the memory, exactly what is and isn’t reclaimed depends somewhat on your exact usage pattern.
The B+Tree’s which are the primary data structure for TDB, the default database used in Fuseki, does not reclaim the space. It is potentially subject fragmentation as well so memory used tends to grow over time. The node table portion of the database, the mapping from RDF terms to internal database identifiers is a sequential data structure that will only ever grow over time. It is also worth noting that many of the data structures are backed by memory mapped files which are off-heap and subject to the vagaries of how your OS handles this.
Additionally, if you place Fuseki under continuous load TDB maybe blocked from writing the in memory journal back to disk which can cause back to grow unbounded overtime and prevent memory being reclaimed. Adding occasional pauses between operations can help to alleviate this.
As Lorenz notes for this kind of use case you may not need Fuseki at all and could simply drive TDB programmatically instead.
As a general point creating a fresh database rather than reusing an existing one will much more efficiently use memory. However, if you’re running on Windows then there is a known OS specific JVM bug that can cause memory mapped files to not be properly deleted until after the process exits.
Rob
On 06/01/2017 12:23, "Janda, Radim" <ra...@reporters.cz> wrote:
Hello Lorenz,
yes I meant delete data from Fuseki using DELETE command.
We have version 2.4 installed.
We use two types of queries:
1. Insert new triples based on existing triples rdf model (insert sparql)
2. Find some results in the data (select sparql)
Thanks
Radim
On Fri, Jan 6, 2017 at 1:04 PM, Lorenz B. <
buehmann@informatik.uni-leipzig.de> wrote:
> Hello Radim,
>
> just to avoid confusion, with "Delete whole Fuseki" you mean the data
> loaded into Fuseki, right?
>
> Which Fuseki version do you use?
>
> What kind of transformation do you do? I'm asking because I'm wondering
> if it's necessary to use Fuseki.
>
>
>
> Cheers,
> Lorenz
>
> > Hello,
> > We use Jena Fuseki to process a lot of small data sets.
> >
> > It works in the following way:
> > 1. Delete whole Fuseki (using DELETE command)
> > 2. Load data to Fuseki (using INSERT)
> > 3. Tranform data and create output (sparql called from Python)
> > 4. ad 1)2)3 .... delete Fuseki and Transform another data set
> >
> > We have found out that memory is not released after delete in Fuseki.
> > That means we have lack of memory after some data sets are transformed.
> > Actually we restart Fuseki server after some number of data sets but we
> > are looking for the better solution.
> >
> > Can you please help us with memory releasing?
> >
> > Many thanks
> >
> > Radim
> >
> --
> Lorenz Bühmann
> AKSW group, University of Leipzig
> Group: http://aksw.org - semantic web research center
>
>
Re: Fuseki - how to release memory
Posted by "Janda, Radim" <ra...@reporters.cz>.
Hello Lorenz,
yes I meant delete data from Fuseki using DELETE command.
We have version 2.4 installed.
We use two types of queries:
1. Insert new triples based on existing triples rdf model (insert sparql)
2. Find some results in the data (select sparql)
Thanks
Radim
On Fri, Jan 6, 2017 at 1:04 PM, Lorenz B. <
buehmann@informatik.uni-leipzig.de> wrote:
> Hello Radim,
>
> just to avoid confusion, with "Delete whole Fuseki" you mean the data
> loaded into Fuseki, right?
>
> Which Fuseki version do you use?
>
> What kind of transformation do you do? I'm asking because I'm wondering
> if it's necessary to use Fuseki.
>
>
>
> Cheers,
> Lorenz
>
> > Hello,
> > We use Jena Fuseki to process a lot of small data sets.
> >
> > It works in the following way:
> > 1. Delete whole Fuseki (using DELETE command)
> > 2. Load data to Fuseki (using INSERT)
> > 3. Tranform data and create output (sparql called from Python)
> > 4. ad 1)2)3 .... delete Fuseki and Transform another data set
> >
> > We have found out that memory is not released after delete in Fuseki.
> > That means we have lack of memory after some data sets are transformed.
> > Actually we restart Fuseki server after some number of data sets but we
> > are looking for the better solution.
> >
> > Can you please help us with memory releasing?
> >
> > Many thanks
> >
> > Radim
> >
> --
> Lorenz Bühmann
> AKSW group, University of Leipzig
> Group: http://aksw.org - semantic web research center
>
>
Re: Fuseki - how to release memory
Posted by "Lorenz B." <bu...@informatik.uni-leipzig.de>.
Hello Radim,
just to avoid confusion, with "Delete whole Fuseki" you mean the data
loaded into Fuseki, right?
Which Fuseki version do you use?
What kind of transformation do you do? I'm asking because I'm wondering
if it's necessary to use Fuseki.
Cheers,
Lorenz
> Hello,
> We use Jena Fuseki to process a lot of small data sets.
>
> It works in the following way:
> 1. Delete whole Fuseki (using DELETE command)
> 2. Load data to Fuseki (using INSERT)
> 3. Tranform data and create output (sparql called from Python)
> 4. ad 1)2)3 .... delete Fuseki and Transform another data set
>
> We have found out that memory is not released after delete in Fuseki.
> That means we have lack of memory after some data sets are transformed.
> Actually we restart Fuseki server after some number of data sets but we
> are looking for the better solution.
>
> Can you please help us with memory releasing?
>
> Many thanks
>
> Radim
>
--
Lorenz Bühmann
AKSW group, University of Leipzig
Group: http://aksw.org - semantic web research center