You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Kieron Taylor <kt...@ebi.ac.uk> on 2017/09/28 09:39:01 UTC
Memory only fuseki dying in a big heap
Hi everyone,
I'm trying to use Fuseki as a temporary memory-only server, i.e. load RDF into memory, run queries, dispose of server.
My testing was going really well until I tried to take it from development on laptop to a compute farm.
JVM 1.8.0_112-b15
Fuseki version 3.4.0
Redhat enterprise 7 via LSF
Server invocation: java --Xmx24GB -Xms24GB -jar fuseki-server.jar --update --port 3355 --mem /test
I load my data (totalling 23 million triples across tens of files) using the s-put utilty into two graphs, and with time and progress depending on how much heap I have allocated (14 GB up to 40 GB), it loads for a while and then explodes. See below for a sample of the whole error.
The perplexing part is that I cannot see any sign of an error to trigger the dump or predict when it will die. If I do not set -Xms to the same as the -Xmx parameter, it dies within ten seconds of starting to load (where loading should take 30 minutes or more). If I give it loads of heap, the crash seems to occur around it receiving its first SPARQL query after the data is loaded. The client (calling s-post) sees generic_request.rb:206:in `copy_stream': Broken pipe - sendfile (Errno::EPIPE), which I infer to mean that the server has gone away mid-request.
I have tried the following so far:
1. Add heap
2. Change JVM to another Java 8 release
3. Turn up Fuseki logging - No debug messages obviously indicate an error prior to the crash
Can anybody recommend a course of action to diagnose and fix my issue?
Regards,
Kieron
----------------- thread dump-----------------------
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):
"qtp596910004-172" #172 prio=5 os_prio=0 tid=0x00002b9c54002000 nid=0xcc67 waiting on condition [0x00002b9bd3a7f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004659a2800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
"qtp596910004-41" #41 prio=5 os_prio=0 tid=0x00002b9c64001000 nid=0xb393 runnable [0x00002b9bd3c2d000]
java.lang.Thread.State: RUNNABLE
..... lots more threads
"VM Thread" os_prio=0 tid=0x00002b9b3c3c5800 nid=0xd1da runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00002b9b3c01f000 nid=0xd1c0 runnable
.... more GC
"VM Periodic Task Thread" os_prio=0 tid=0x00002b9b3c432800 nid=0xd1ff waiting on condition
JNI global references: 499
Heap
PSYoungGen total 3185664K, used 1577280K [0x000000069c580000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1592832K, 99% used [0x000000069c580000,0x00000006fc9d0308,0x00000006fd900000)
from space 1592832K, 0% used [0x00000006fd900000,0x00000006fd900000,0x000000075ec80000)
to space 1592832K, 0% used [0x000000075ec80000,0x000000075ec80000,0x00000007c0000000)
ParOldGen total 9557504K, used 9557170K [0x0000000455000000, 0x000000069c580000, 0x000000069c580000)
object space 9557504K, 99% used [0x0000000455000000,0x000000069c52c950,0x000000069c580000)
Metaspace used 27217K, capacity 27644K, committed 28032K, reserved 1073152K
class space used 3508K, capacity 3636K, committed 3712K, reserved 1048576K
Kieron Taylor PhD.
Ensembl Developer
EMBL, European Bioinformatics Institute
Re: Memory only fuseki dying in a big heap
Posted by Kieron Taylor <kt...@ebi.ac.uk>.
Thanks Andy, I appreciate your insight.
> It's -Xmx, not --Xmx
A typo from reconstituting the command line from wrapper code. It's definitely allocating heap judging by LSF reports on the job.
> I'd have thought 24G is plenty for 23 million triples unless there are many very large literals.
I was averaging about 1GB per million triples during testing, but it runs on less up to a point.
>> I have tried the following so far:
>> 1. Add heap
>> 2. Change JVM to another Java 8 release
>> 3. Turn up Fuseki logging - No debug messages obviously indicate an error prior to the crash
>> Can anybody recommend a course of action to diagnose and fix my issue?
>
> I hate to say it but it smells a bit like a hardware fault (or JVM fault?), especially the unpredictability. Anything software is usually reasonably predictable.
Thanks for this. The main thing is that it's not a common occurence for Jena users. I had hoped it was related to some kind of memory allocation error around sort/search buffers or similar, but that is more in the territory of Virtuoso.
I am leaning toward the process being ungracefully terminated by the wrapping code, rather than it killing itself from within Fuseki. This is a good thing! Fuseki barked loudest which made me believe it was the issue, but I have good indications that Fuseki is working properly.
Thanks again,
Kieron
Re: Memory only fuseki dying in a big heap
Posted by Andy Seaborne <an...@apache.org>.
Hi Kieron,
On 28/09/17 10:39, Kieron Taylor wrote:
> Hi everyone,
>
> I'm trying to use Fuseki as a temporary memory-only server, i.e. load RDF into memory, run queries, dispose of server.
>
> My testing was going really well until I tried to take it from development on laptop to a compute farm.
>
> JVM 1.8.0_112-b15
> Fuseki version 3.4.0
> Redhat enterprise 7 via LSF
>
> Server invocation: java --Xmx24GB -Xms24GB -jar fuseki-server.jar --update --port 3355 --mem /test
^^^^^^
It's -Xmx, not --Xmx
I thought that caused an error - but if it doesn't the max heap isn't
set, which explain why -Xms is needed.
I'd have thought 24G is plenty for 23 million triples unless there are
many very large literals.
>
> I load my data (totalling 23 million triples across tens of files) using the s-put utilty into two graphs, and with time and progress depending on how much heap I have allocated (14 GB up to 40 GB), it loads for a while and then explodes. See below for a sample of the whole error.
>
> The perplexing part is that I cannot see any sign of an error to trigger the dump or predict when it will die. If I do not set -Xms to the same as the -Xmx parameter, it dies within ten seconds of starting to load (where loading should take 30 minutes or more). If I give it loads of heap, the crash seems to occur around it receiving its first SPARQL query after the data is loaded. The client (calling s-post) sees generic_request.rb:206:in `copy_stream': Broken pipe - sendfile (Errno::EPIPE), which I infer to mean that the server has gone away mid-request.
Data is added transactionally so even if a bad update happens the rest
of the data should be safe.
>
> I have tried the following so far:
>
> 1. Add heap
> 2. Change JVM to another Java 8 release
> 3. Turn up Fuseki logging - No debug messages obviously indicate an error prior to the crash
>
> Can anybody recommend a course of action to diagnose and fix my issue?
I hate to say it but it smells a bit like a hardware fault (or JVM
fault?), especially the unpredictability. Anything software is usually
reasonably predictable.
Andy
>
>
> Regards,
>
> Kieron
>
> ----------------- thread dump-----------------------
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):
>
> "qtp596910004-172" #172 prio=5 os_prio=0 tid=0x00002b9c54002000 nid=0xcc67 waiting on condition [0x00002b9bd3a7f000]
> java.lang.Thread.State: WAITING (parking)
This is the webserver (Jetty) waiting for something.
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000004659a2800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:173)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp596910004-41" #41 prio=5 os_prio=0 tid=0x00002b9c64001000 nid=0xb393 runnable [0x00002b9bd3c2d000]
> java.lang.Thread.State: RUNNABLE
> ..... lots more threads
>
> "VM Thread" os_prio=0 tid=0x00002b9b3c3c5800 nid=0xd1da runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00002b9b3c01f000 nid=0xd1c0 runnable
> .... more GC
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00002b9b3c432800 nid=0xd1ff waiting on condition
>
> JNI global references: 499
>
> Heap
> PSYoungGen total 3185664K, used 1577280K [0x000000069c580000, 0x00000007c0000000, 0x00000007c0000000)
> eden space 1592832K, 99% used [0x000000069c580000,0x00000006fc9d0308,0x00000006fd900000)
> from space 1592832K, 0% used [0x00000006fd900000,0x00000006fd900000,0x000000075ec80000)
> to space 1592832K, 0% used [0x000000075ec80000,0x000000075ec80000,0x00000007c0000000)
> ParOldGen total 9557504K, used 9557170K [0x0000000455000000, 0x000000069c580000, 0x000000069c580000)
> object space 9557504K, 99% used [0x0000000455000000,0x000000069c52c950,0x000000069c580000)
> Metaspace used 27217K, capacity 27644K, committed 28032K, reserved 1073152K
> class space used 3508K, capacity 3636K, committed 3712K, reserved 1048576K
I don't see more the 10G being used here.
>
> Kieron Taylor PhD.
> Ensembl Developer
>
> EMBL, European Bioinformatics Institute
>