You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Laura Morales <la...@mail.com> on 2023/07/09 19:33:44 UTC
OOM Killed
I'm running a job that is submitting a lot of queries to a Fuseki server, in parallel. My problem is that Fuseki is OOM-killed and I don't know how to fix this. Some details:
- Fuseki is queried as fast as possible. Queries take around 50-100ms to complete so I think it's serving 10s of queries each second
- Fuseki 4.8. OS is Debian 12 (minimal installation with only OS, Fuseki, no desktop environments, uses only ~100MB of RAM)
- all the queries are read queries. No updates, inserts, or other write queries
- all the queries are over HTTP to the Fuseki endpoint
- database is TDB2 (created with tdb2.tdbloader)
- database contains around 2.5M triples
- the machine has 8GB RAM. I've tried on another PC with 16GB and it completes the job. On 8GB though, it won't
- with -Xmx6G it's killed earlier. With -Xmx2G it's killed later. Either way it's always killed.
Is there anything that I can tweak to avoid Fuseki getting killed? Something that isn't "just buy more RAM".
Thank you
Re: OOM Killed
Posted by Laura Morales <la...@mail.com>.
- database contains around 2.5M triples and is ~4GB in size on disk
Re: OOM Killed
Posted by Laura Morales <la...@mail.com>.
Well, I think the OOM trigger that prompted Linux to kill Fuseki was the fact that I had a very small swap space (1GB). After adding a new swap partition (256GB) I don't see any errors anymore (OOM or heap space).
On my PC with 16GB RAM, it used 2-3GB of swap and took approximately the same amount of time to finish as in my previous tests.
On my PC with 8GB RAM, it used 9-10GB of swap and took significantly longer to finish, ~3h instead of ~2h.
The good news for me is, I guess, that I've found something that works for me. On the other hand I think there is a memory problem with Fuseki because it doesn't feel right when it's using that much RAM for processing read queries in series (not in parallel).
I would still love to know if there are options for forcing Fuseki into managing a given amount of RAM.
Re: OOM Killed
Posted by Andy Seaborne <an...@apache.org>.
On 10/07/2023 12:18, Andy Seaborne wrote:
> Laura, Dave,
>
> This doesn't sound like the same issue but let's see.
>
Sorry for the confusion - these questions are for Laura.
>
> - Is it in a container? If so:
> Is it the container being killed OOM or
> Java throwing an OOM exception?
> Much RAM does the container get? How many threads?
>
> - If not a container, how many CPU Threads are there? How many cores?
>
> - Which form of Fuseki are you using?
>
> what does
> java -XX:+PrintFlagsFinal -version \
> | grep -i 'M..HeapSize`
>
> say?
>
> How are you sending the queries to the server?
>
> On 09/07/2023 20:33, Laura Morales wrote:
>> I'm running a job that is submitting a lot of queries to a Fuseki
>> server, in parallel. My problem is that Fuseki is OOM-killed and I
>> don't know how to fix this. Some details:
>>
>> - Fuseki is queried as fast as possible. Queries take around 50-100ms
>> to complete so I think it's serving 10s of queries each second
>
> Are all the queries about the same amount of work are are some going to
> cause significantly more memory use?
>
> It is quite possible to send queries faster than the server can process
> them - there is little point sending in parallel more than there are
> real CPU threads to service them.
>
> They will interfere and the machine can end up going slower (query of
> queries per second).
>
> I don't know exactly the impact on the GC but I think the JVM delays
> minor GC's when very busy but that pushes it to do major ones earlier.
>
> A thing to try is use less parallelism.
>
>> - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS,
>> Fuseki, no desktop environments, uses only ~100MB of RAM)
>> - all the queries are read queries. No updates, inserts, or other
>> write queries
>> - all the queries are over HTTP to the Fuseki endpoint
>> - database is TDB2 (created with tdb2.tdbloader)
>> - database contains around 2.5M triples
>> - the machine has 8GB RAM. I've tried on another PC with 16GB and it
>> completes the job. On 8GB though, it won't
>> - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later.
>> Either way it's always killed.
>
> Is it getting OOM at random or do certain queries tend to push it over
> he edge?
>
> Is that the machine (container) has 8G RAM and there is no -Xmx setting?
> in that case, default setting applies which is 25% of RAM.
>
> A heap dump to know where the memory is going would be useful.
>
>> Is there anything that I can tweak to avoid Fuseki getting killed?
>> Something that isn't "just buy more RAM".
>> Thank you
Re: OOM Killed
Posted by Dave Reynolds <da...@gmail.com>.
Hi Andy,
On 10/07/2023 12:18, Andy Seaborne wrote:
> Laura, Dave,
>
> This doesn't sound like the same issue but let's see.
It may well be different, if so apologies for causing noise.
> Dave - your situation isn't under high load is it?
We see the process size growth under no load other than metric scrapes.
However, growth seems faster if there's more traffic (faster scrapes) so
expecting that high query load will make it worse. Whether "worse" means
it'll just get to some asymptote faster or actually go higher is unproven.
> - Is it in a container? If so:
The original problem was in a container. But as I've said we can
reproduce the process growth on bare metal (i.e. local desktop).
> Is it the container being killed OOM or
> Java throwing an OOM exception?
For the original problem it's the container being OOM killed, no java
exception.
For local tests both on container and bare metal we've just been looking
at the process size growth, haven't run it long enough to reach OOM
state on the size of machine I'm using then I doubt it will on a
timescale I can wait for.
> Much RAM does the container get? How many threads?
For the original problem the container had no memory limit other than
machine total of 4GB. No constraints set on threads.
> - If not a container, how many CPU Threads are there? How many cores?
For local tests 6 cores, should mean 12 CPU threads but checking just
now I suspect hyperthreading isn't working on my current install so call
it 6 of both.
> - Which form of Fuseki are you using?
fuseki-server
> what does
> java -XX:+PrintFlagsFinal -version \
> | grep -i 'M..HeapSize`
>
> say?
E.g. in the container:
size_t ErgoHeapSizeLimit = 0
{product} {default}
size_t HeapSizePerGCThread = 43620760
{product} {default}
size_t InitialHeapSize = 65011712
{product} {ergonomic}
size_t LargePageHeapSizeThreshold = 134217728
{product} {default}
size_t MaxHeapSize = 1019215872
{product} {ergonomic}
size_t MinHeapSize = 8388608
{product} {ergonomic}
uintx NonNMethodCodeHeapSize = 5826188
{pd product} {ergonomic}
uintx NonProfiledCodeHeapSize = 122916026
{pd product} {ergonomic}
uintx ProfiledCodeHeapSize = 122916026
{pd product} {ergonomic}
size_t SoftMaxHeapSize = 1019215872
{manageable} {ergonomic}
But we are overriding the MaxHeapSize with the -Xmx flag in the actual
running process.
> How are you sending the queries to the server?
For the original problems this occurred on a system with no queries at
all just the metrics scraping. The /$/ping point was getting checked by
a healthcheck monitoring tool (sensu) and /$/metrics by prometheus.
On the local bare metal checks where we can reproduce process growth at
least medium term we are just checking those by curl in a watch loop.
We've made some process at our end, and update on that back on the
previous thread rather than further confuse this one.
Dave
> On 09/07/2023 20:33, Laura Morales wrote:
>> I'm running a job that is submitting a lot of queries to a Fuseki
>> server, in parallel. My problem is that Fuseki is OOM-killed and I
>> don't know how to fix this. Some details:
>>
>> - Fuseki is queried as fast as possible. Queries take around 50-100ms
>> to complete so I think it's serving 10s of queries each second
>
> Are all the queries about the same amount of work are are some going to
> cause significantly more memory use?
>
> It is quite possible to send queries faster than the server can process
> them - there is little point sending in parallel more than there are
> real CPU threads to service them.
>
> They will interfere and the machine can end up going slower (query of
> queries per second).
>
> I don't know exactly the impact on the GC but I think the JVM delays
> minor GC's when very busy but that pushes it to do major ones earlier.
>
> A thing to try is use less parallelism.
>
>> - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS,
>> Fuseki, no desktop environments, uses only ~100MB of RAM)
>> - all the queries are read queries. No updates, inserts, or other
>> write queries
>> - all the queries are over HTTP to the Fuseki endpoint
>> - database is TDB2 (created with tdb2.tdbloader)
>> - database contains around 2.5M triples
>> - the machine has 8GB RAM. I've tried on another PC with 16GB and it
>> completes the job. On 8GB though, it won't
>> - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later.
>> Either way it's always killed.
>
> Is it getting OOM at random or do certain queries tend to push it over
> he edge?
>
> Is that the machine (container) has 8G RAM and there is no -Xmx setting?
> in that case, default setting applies which is 25% of RAM.
>
> A heap dump to know where the memory is going would be useful.
>
>> Is there anything that I can tweak to avoid Fuseki getting killed?
>> Something that isn't "just buy more RAM".
>> Thank you
Re: OOM Killed
Posted by Laura Morales <la...@mail.com>.
> Could you try Java17?
I did try Java 17 (default jre available from Debian) but I didn't notice any difference. I only ran 2 tests with Java 17 though, because the job takes a long time to fail/finish. I only noticed a significant difference when tweaking the GC options.
However I did another test on a 3rd PC, with the same exact software setup (Debian 12, Java 17, Fuseki 4.9.0) but completely different hardware (i5 7th-gen 4C4T, 16GB RAM, NVMe PCIe-3 x4) and it completed in ~30m (instead of ~2h) and the max RAM usage for the Java process was 11.3GB *without* tuning any GC or Fuseki options.
I don't know what to make of this. Maybe it's the hardware after all, and not Fuseki. Or maybe the GC has more time for doing its thing with a NVMe drive instead of SATA/USB. However 11GB still seems very high. As far as I know, each HTTP request should be short lived, and all the memory of every HTTP request should be freed (that's why it's strange that I see the memory grow over time).
Re: OOM Killed
Posted by Andy Seaborne <an...@apache.org>.
Hi Laura,
It hadn't occurred to me that the GC choice might be involved.
Also, I though G1 was the default GC but it seems at java11 it isn't
that simple. It's build dependent.
I use the Ubuntu build of OpenJDK.
Java11 has Shenandoah
Java17 has G1 -- and I think java21 will be G1.
Could you try Java17?
And Jena will be moving to require java17. The project supports 2 LTS
and Java21 in September is LTS.
Andy
On 14/07/2023 08:54, Laura Morales wrote:
>> Have you tried different garbage collectors?
>
> WOAH I didn't even consider that before you mentioned it! I did this
>
> JVM_ARGS="-XX:+UseSerialGC -Xmx4G" ./fuseki-server ...
>
> and RAM usage of the java process peaked at 12GB
>
> $ cat /proc/108344/status | grep VmHWM
> VmHWM: 11916368 kB
>
> Unfortunately I'm not at all familiar with Java garbage collectors. I don't understand why this option would use 1/3 less RAM than the default GC.
> What other options are available for a more aggressive GC? I'm more interested in reducing RAM usage than raw query performance.
Re: OOM Killed
Posted by Laura Morales <la...@mail.com>.
> Have you tried different garbage collectors?
WOAH I didn't even consider that before you mentioned it! I did this
JVM_ARGS="-XX:+UseSerialGC -Xmx4G" ./fuseki-server ...
and RAM usage of the java process peaked at 12GB
$ cat /proc/108344/status | grep VmHWM
VmHWM: 11916368 kB
Unfortunately I'm not at all familiar with Java garbage collectors. I don't understand why this option would use 1/3 less RAM than the default GC.
What other options are available for a more aggressive GC? I'm more interested in reducing RAM usage than raw query performance.
Re: OOM Killed
Posted by Andy Seaborne <an...@apache.org>.
> size_t MaxHeapSize = 4175429632
{product} {ergonomic}
> size_t ShenandoahSoftMaxHeapSize = 0
{manageable} {default}
Have you tried different garbage collectors?
Shenandoah does its work in parallel so that may be heap in use not by
the queries.
Andy
On 11/07/2023 10:39, Laura Morales wrote:
> I have tried to do some testing but I cannot get a definitive answer about what works and what not, because there are so many variables. Also it doesn't fail right away, but after 1h to 1.5h, so I've done fewer than a dozen tests actually. I'm sorry that I can't be more precise.
>
> The two PCs that I've got are an i3 4th-gen 2C4T with 8GB RAM, and an i7 4th-gen 2C4T with 16GB RAM. The database is stored in a 1TB USB3.0 SSD (which I move between the two PCs). Either way, the only component that seems to make a difference is the amount of RAM. They are physical machines (bare metal) and not containers.
> I'm running "fuseki-server" binary, downloaded from the binaries distribution, "./fuseki-server --loc=database --port=7000 --localhost /query".
> I query Fuseki (4.8) over HTTP from a script that runs on the same PC. The script uses <100MB of RAM. All queries are "read" (no writes) and actually very basic. Either "DESCRIBE <node>" or "SELECT" that selects a node and follows a couple of links 3 or 4 levels deep at most. Fuseki answers them very quickly, <5ms, but occasionally it takes 50-100ms or rarely a couple of seconds (probably because of garbage collection?). The queries are always the same, run over and over across the dataset. The dataset contains approximately 200K nodes, 2.5M triples, 4GB disk size. Most nodes have, among their properties, 3 that contain long strings, approximately 20KB-50KB combined, per node. I don't query those properties directly, but when I "DESCRIBE" a node they are retrieved. Fuseki is very fast, but these strings may contribute to the high memory load (I'm only guessing). I cannot identify any particular query as the offender, since it's the same bunch of queries run over and over at max speed (ie. one after the other with no wait time). It's pretty much the same amount of work for every query, and it works perfectly fine until it consumes all the RAM and SWAP.
>
> The only configuration that worked for me (ie. it completed the job) is -Xmx4G and no parallelism at all (one request after the other in series) on 16GB of RAM (it used up all the RAM available). It seems strange to me that it needs so much RAM even when all the requests are serialized. Querying in parallel with 16 or 32 threads doesn't seem to make much of a difference to Fuseki, other than the even higher memory consumption (Fuseki answers all queries very quickly in milliseconds, until it runs out of memory).
> The memory growth is not instantaneous, and is not linear. I can see RAM usage fluctuate by a 3-4GB range (for example between 3GB and 7GB). But the trend is to use more and more memory. For example before crashing it would fluctuate between 10GB and 15GB.
> If I increase -Xmx over 4GB, Fuseki is eventually OOM killed by the kernel. Below 4GB, Fuseki crashes with a heap error like this (both cases fail well after 1h of work):
>
> 10:03:21 WARN QueuedThreadPool :: Job failed
> java.lang.OutOfMemoryError: Java heap space
> 10:03:21 WARN Fuseki :: [152378] RC = 500 : Java heap space: failed reallocation of scalar replaced objects
> java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
> 10:03:21 INFO Fuseki :: [152378] 500 Server Error (48.115 s)
> 10:03:23 WARN AbstractConnector :: Accept Failure
> java.lang.OutOfMemoryError: Java heap space
> 10:04:08 WARN QueuedThreadPool :: Job failed
> java.lang.OutOfMemoryError: Java heap space
> 10:04:08 WARN QueuedThreadPool :: Job failed
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "HttpClient-2-SelectorManager" java.lang.OutOfMemoryError: Java heap space
>
> On 8GB RAM it always fails for me, -Xmx4G or more is OOM-killed, whereas less ends up with a heap error.
> The output of "java -XX:+PrintFlagsFinal -version | grep -i "M..HeapSize"" is
>
> size_t MaxHeapSize = 4175429632 {product} {ergonomic}
> size_t ShenandoahSoftMaxHeapSize = 0 {manageable} {default}
> openjdk version "11.0.18" 2023-01-17
> OpenJDK Runtime Environment (build 11.0.18+10-post-Debian-1deb11u1)
> OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Debian-1deb11u1, mixed mode, sharing)
>
> I've also tried with OpenJDK 17, same results.
> I tried Fuseki 3.17 too but I was getting other JSON-LD errors (probably related to an old JSON-LD library) so I didn't test further.
>
> I know that I don't have the latest and greatest hardware, but I think my database is very small and I feel like Fuseki should not be using 16GB RAM when running a lot of simple queries in series (not in parallel).
> One thing that I want to try, but so far haven't, is to restart Fuseki halfway through the job.
>
>
>
>> Sent: Monday, July 10, 2023 at 1:18 PM
>> From: "Andy Seaborne" <an...@apache.org>
>> To: users@jena.apache.org
>> Subject: Re: OOM Killed
>>
>> Laura, Dave,
>>
>> This doesn't sound like the same issue but let's see.
>>
>> Dave - your situation isn't under high load is it?
>>
>> - Is it in a container? If so:
>> Is it the container being killed OOM or
>> Java throwing an OOM exception?
>> Much RAM does the container get? How many threads?
>>
>> - If not a container, how many CPU Threads are there? How many cores?
>>
>> - Which form of Fuseki are you using?
>>
>> what does
>> java -XX:+PrintFlagsFinal -version \
>> | grep -i 'M..HeapSize`
>>
>> say?
>>
>> How are you sending the queries to the server?
>>
>> On 09/07/2023 20:33, Laura Morales wrote:
>>> I'm running a job that is submitting a lot of queries to a Fuseki server, in parallel. My problem is that Fuseki is OOM-killed and I don't know how to fix this. Some details:
>>>
>>> - Fuseki is queried as fast as possible. Queries take around 50-100ms to complete so I think it's serving 10s of queries each second
>>
>> Are all the queries about the same amount of work are are some going to
>> cause significantly more memory use?
>>
>> It is quite possible to send queries faster than the server can process
>> them - there is little point sending in parallel more than there are
>> real CPU threads to service them.
>>
>> They will interfere and the machine can end up going slower (query of
>> queries per second).
>>
>> I don't know exactly the impact on the GC but I think the JVM delays
>> minor GC's when very busy but that pushes it to do major ones earlier.
>>
>> A thing to try is use less parallelism.
>>
>>> - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS, Fuseki, no desktop environments, uses only ~100MB of RAM)
>>> - all the queries are read queries. No updates, inserts, or other write queries
>>> - all the queries are over HTTP to the Fuseki endpoint
>>> - database is TDB2 (created with tdb2.tdbloader)
>>> - database contains around 2.5M triples
>>> - the machine has 8GB RAM. I've tried on another PC with 16GB and it completes the job. On 8GB though, it won't
>>> - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later. Either way it's always killed.
>>
>> Is it getting OOM at random or do certain queries tend to push it over
>> he edge?
>>
>> Is that the machine (container) has 8G RAM and there is no -Xmx setting?
>> in that case, default setting applies which is 25% of RAM.
>>
>> A heap dump to know where the memory is going would be useful.
>>
>>> Is there anything that I can tweak to avoid Fuseki getting killed? Something that isn't "just buy more RAM".
>>> Thank you
>>
Re: OOM Killed
Posted by Laura Morales <la...@mail.com>.
I've started Fuseki
./fuseki-server --loc=database --localhost --port=7000 /query
and then I've started my script, with 16 threads (each one making a request to Fuseki). I did not set -Xmx, but I saw in htop that 4GB was chosen automatically. I did this on my PC with 16GB RAM, 2C4T CPU. When I was seeing Fuseki using around 12GB of memory, I would pause my script, restart Fuseki, and resume my script. I did this a couple of times and the job completed without issues (no OOM and no crashes). It took ~2h.
I don't know if this adds any useful information.
Re: OOM Killed
Posted by Laura Morales <la...@mail.com>.
I have tried to do some testing but I cannot get a definitive answer about what works and what not, because there are so many variables. Also it doesn't fail right away, but after 1h to 1.5h, so I've done fewer than a dozen tests actually. I'm sorry that I can't be more precise.
The two PCs that I've got are an i3 4th-gen 2C4T with 8GB RAM, and an i7 4th-gen 2C4T with 16GB RAM. The database is stored in a 1TB USB3.0 SSD (which I move between the two PCs). Either way, the only component that seems to make a difference is the amount of RAM. They are physical machines (bare metal) and not containers.
I'm running "fuseki-server" binary, downloaded from the binaries distribution, "./fuseki-server --loc=database --port=7000 --localhost /query".
I query Fuseki (4.8) over HTTP from a script that runs on the same PC. The script uses <100MB of RAM. All queries are "read" (no writes) and actually very basic. Either "DESCRIBE <node>" or "SELECT" that selects a node and follows a couple of links 3 or 4 levels deep at most. Fuseki answers them very quickly, <5ms, but occasionally it takes 50-100ms or rarely a couple of seconds (probably because of garbage collection?). The queries are always the same, run over and over across the dataset. The dataset contains approximately 200K nodes, 2.5M triples, 4GB disk size. Most nodes have, among their properties, 3 that contain long strings, approximately 20KB-50KB combined, per node. I don't query those properties directly, but when I "DESCRIBE" a node they are retrieved. Fuseki is very fast, but these strings may contribute to the high memory load (I'm only guessing). I cannot identify any particular query as the offender, since it's the same bunch of queries run over and over at max speed (ie. one after the other with no wait time). It's pretty much the same amount of work for every query, and it works perfectly fine until it consumes all the RAM and SWAP.
The only configuration that worked for me (ie. it completed the job) is -Xmx4G and no parallelism at all (one request after the other in series) on 16GB of RAM (it used up all the RAM available). It seems strange to me that it needs so much RAM even when all the requests are serialized. Querying in parallel with 16 or 32 threads doesn't seem to make much of a difference to Fuseki, other than the even higher memory consumption (Fuseki answers all queries very quickly in milliseconds, until it runs out of memory).
The memory growth is not instantaneous, and is not linear. I can see RAM usage fluctuate by a 3-4GB range (for example between 3GB and 7GB). But the trend is to use more and more memory. For example before crashing it would fluctuate between 10GB and 15GB.
If I increase -Xmx over 4GB, Fuseki is eventually OOM killed by the kernel. Below 4GB, Fuseki crashes with a heap error like this (both cases fail well after 1h of work):
10:03:21 WARN QueuedThreadPool :: Job failed
java.lang.OutOfMemoryError: Java heap space
10:03:21 WARN Fuseki :: [152378] RC = 500 : Java heap space: failed reallocation of scalar replaced objects
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
10:03:21 INFO Fuseki :: [152378] 500 Server Error (48.115 s)
10:03:23 WARN AbstractConnector :: Accept Failure
java.lang.OutOfMemoryError: Java heap space
10:04:08 WARN QueuedThreadPool :: Job failed
java.lang.OutOfMemoryError: Java heap space
10:04:08 WARN QueuedThreadPool :: Job failed
java.lang.OutOfMemoryError: Java heap space
Exception in thread "HttpClient-2-SelectorManager" java.lang.OutOfMemoryError: Java heap space
On 8GB RAM it always fails for me, -Xmx4G or more is OOM-killed, whereas less ends up with a heap error.
The output of "java -XX:+PrintFlagsFinal -version | grep -i "M..HeapSize"" is
size_t MaxHeapSize = 4175429632 {product} {ergonomic}
size_t ShenandoahSoftMaxHeapSize = 0 {manageable} {default}
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+10-post-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Debian-1deb11u1, mixed mode, sharing)
I've also tried with OpenJDK 17, same results.
I tried Fuseki 3.17 too but I was getting other JSON-LD errors (probably related to an old JSON-LD library) so I didn't test further.
I know that I don't have the latest and greatest hardware, but I think my database is very small and I feel like Fuseki should not be using 16GB RAM when running a lot of simple queries in series (not in parallel).
One thing that I want to try, but so far haven't, is to restart Fuseki halfway through the job.
> Sent: Monday, July 10, 2023 at 1:18 PM
> From: "Andy Seaborne" <an...@apache.org>
> To: users@jena.apache.org
> Subject: Re: OOM Killed
>
> Laura, Dave,
>
> This doesn't sound like the same issue but let's see.
>
> Dave - your situation isn't under high load is it?
>
> - Is it in a container? If so:
> Is it the container being killed OOM or
> Java throwing an OOM exception?
> Much RAM does the container get? How many threads?
>
> - If not a container, how many CPU Threads are there? How many cores?
>
> - Which form of Fuseki are you using?
>
> what does
> java -XX:+PrintFlagsFinal -version \
> | grep -i 'M..HeapSize`
>
> say?
>
> How are you sending the queries to the server?
>
> On 09/07/2023 20:33, Laura Morales wrote:
> > I'm running a job that is submitting a lot of queries to a Fuseki server, in parallel. My problem is that Fuseki is OOM-killed and I don't know how to fix this. Some details:
> >
> > - Fuseki is queried as fast as possible. Queries take around 50-100ms to complete so I think it's serving 10s of queries each second
>
> Are all the queries about the same amount of work are are some going to
> cause significantly more memory use?
>
> It is quite possible to send queries faster than the server can process
> them - there is little point sending in parallel more than there are
> real CPU threads to service them.
>
> They will interfere and the machine can end up going slower (query of
> queries per second).
>
> I don't know exactly the impact on the GC but I think the JVM delays
> minor GC's when very busy but that pushes it to do major ones earlier.
>
> A thing to try is use less parallelism.
>
> > - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS, Fuseki, no desktop environments, uses only ~100MB of RAM)
> > - all the queries are read queries. No updates, inserts, or other write queries
> > - all the queries are over HTTP to the Fuseki endpoint
> > - database is TDB2 (created with tdb2.tdbloader)
> > - database contains around 2.5M triples
> > - the machine has 8GB RAM. I've tried on another PC with 16GB and it completes the job. On 8GB though, it won't
> > - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later. Either way it's always killed.
>
> Is it getting OOM at random or do certain queries tend to push it over
> he edge?
>
> Is that the machine (container) has 8G RAM and there is no -Xmx setting?
> in that case, default setting applies which is 25% of RAM.
>
> A heap dump to know where the memory is going would be useful.
>
> > Is there anything that I can tweak to avoid Fuseki getting killed? Something that isn't "just buy more RAM".
> > Thank you
>
Re: OOM Killed
Posted by Andy Seaborne <an...@apache.org>.
Laura, Dave,
This doesn't sound like the same issue but let's see.
Dave - your situation isn't under high load is it?
- Is it in a container? If so:
Is it the container being killed OOM or
Java throwing an OOM exception?
Much RAM does the container get? How many threads?
- If not a container, how many CPU Threads are there? How many cores?
- Which form of Fuseki are you using?
what does
java -XX:+PrintFlagsFinal -version \
| grep -i 'M..HeapSize`
say?
How are you sending the queries to the server?
On 09/07/2023 20:33, Laura Morales wrote:
> I'm running a job that is submitting a lot of queries to a Fuseki server, in parallel. My problem is that Fuseki is OOM-killed and I don't know how to fix this. Some details:
>
> - Fuseki is queried as fast as possible. Queries take around 50-100ms to complete so I think it's serving 10s of queries each second
Are all the queries about the same amount of work are are some going to
cause significantly more memory use?
It is quite possible to send queries faster than the server can process
them - there is little point sending in parallel more than there are
real CPU threads to service them.
They will interfere and the machine can end up going slower (query of
queries per second).
I don't know exactly the impact on the GC but I think the JVM delays
minor GC's when very busy but that pushes it to do major ones earlier.
A thing to try is use less parallelism.
> - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS, Fuseki, no desktop environments, uses only ~100MB of RAM)
> - all the queries are read queries. No updates, inserts, or other write queries
> - all the queries are over HTTP to the Fuseki endpoint
> - database is TDB2 (created with tdb2.tdbloader)
> - database contains around 2.5M triples
> - the machine has 8GB RAM. I've tried on another PC with 16GB and it completes the job. On 8GB though, it won't
> - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later. Either way it's always killed.
Is it getting OOM at random or do certain queries tend to push it over
he edge?
Is that the machine (container) has 8G RAM and there is no -Xmx setting?
in that case, default setting applies which is 25% of RAM.
A heap dump to know where the memory is going would be useful.
> Is there anything that I can tweak to avoid Fuseki getting killed? Something that isn't "just buy more RAM".
> Thank you
Re: OOM Killed
Posted by "Rob @ DNR" <rv...@dotnetrdf.org>.
While we appreciate that this unresolved memory issue is painful for users, I would strongly emphasise that the project DOES NOT recommend people use outdated versions of Jena.
3.17.0 released in November 2020 which makes it 2.5 years old at this point. There have been lots of security, performance and correctness fixes that have happened in that time.
Rob
From: Dave Reynolds <da...@gmail.com>
Date: Monday, 10 July 2023 at 08:36
To: users@jena.apache.org <us...@jena.apache.org>
Subject: Re: OOM Killed
There is an issue with memory growth in fuseki, though it's growth
outside of normal java heap and non-heap space.
See https://www.mail-archive.com/users@jena.apache.org/msg20362.html
For that scale of data and scale of machine suggest setting the heap
smaller -Xmx1G or -Xmx500M. Empirically the process growth seems to
largely levels off at around 4x the given heap size (though this very
much depends on the usage model and have no clear explanation for this).
You might also try -XX:MaxDirectMemorySize=1G or less though exactly
what size to set will depend on how much data is involved in your
queries. If the process dies with an exception about unable to allocate
new direct memory then increase it.
If this is not a public service liable to security issues and you are
able to use a 3.17.0 (or earlier) version of fuseki then those are not
subject to this growth issue. Or at least not to the version of the
issue that we are seeing in our own usage.
Dave
On 09/07/2023 20:33, Laura Morales wrote:
> I'm running a job that is submitting a lot of queries to a Fuseki server, in parallel. My problem is that Fuseki is OOM-killed and I don't know how to fix this. Some details:
>
> - Fuseki is queried as fast as possible. Queries take around 50-100ms to complete so I think it's serving 10s of queries each second
> - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS, Fuseki, no desktop environments, uses only ~100MB of RAM)
> - all the queries are read queries. No updates, inserts, or other write queries
> - all the queries are over HTTP to the Fuseki endpoint
> - database is TDB2 (created with tdb2.tdbloader)
> - database contains around 2.5M triples
> - the machine has 8GB RAM. I've tried on another PC with 16GB and it completes the job. On 8GB though, it won't
> - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later. Either way it's always killed.
>
> Is there anything that I can tweak to avoid Fuseki getting killed? Something that isn't "just buy more RAM".
> Thank you
Re: OOM Killed
Posted by Dave Reynolds <da...@gmail.com>.
There is an issue with memory growth in fuseki, though it's growth
outside of normal java heap and non-heap space.
See https://www.mail-archive.com/users@jena.apache.org/msg20362.html
For that scale of data and scale of machine suggest setting the heap
smaller -Xmx1G or -Xmx500M. Empirically the process growth seems to
largely levels off at around 4x the given heap size (though this very
much depends on the usage model and have no clear explanation for this).
You might also try -XX:MaxDirectMemorySize=1G or less though exactly
what size to set will depend on how much data is involved in your
queries. If the process dies with an exception about unable to allocate
new direct memory then increase it.
If this is not a public service liable to security issues and you are
able to use a 3.17.0 (or earlier) version of fuseki then those are not
subject to this growth issue. Or at least not to the version of the
issue that we are seeing in our own usage.
Dave
On 09/07/2023 20:33, Laura Morales wrote:
> I'm running a job that is submitting a lot of queries to a Fuseki server, in parallel. My problem is that Fuseki is OOM-killed and I don't know how to fix this. Some details:
>
> - Fuseki is queried as fast as possible. Queries take around 50-100ms to complete so I think it's serving 10s of queries each second
> - Fuseki 4.8. OS is Debian 12 (minimal installation with only OS, Fuseki, no desktop environments, uses only ~100MB of RAM)
> - all the queries are read queries. No updates, inserts, or other write queries
> - all the queries are over HTTP to the Fuseki endpoint
> - database is TDB2 (created with tdb2.tdbloader)
> - database contains around 2.5M triples
> - the machine has 8GB RAM. I've tried on another PC with 16GB and it completes the job. On 8GB though, it won't
> - with -Xmx6G it's killed earlier. With -Xmx2G it's killed later. Either way it's always killed.
>
> Is there anything that I can tweak to avoid Fuseki getting killed? Something that isn't "just buy more RAM".
> Thank you