You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Daniel Hobi <d....@gmx.ch> on 2013/03/07 14:08:08 UTC
JVM hangs during startup (indexing) (2)
Hi there
Some updates on our issue started here:
http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C201
30115135117.186960%40gmx.net%3E
We're still facing the JVM freeze (possibly during minor gcs) even if...
...we updated Jackrabbit from 2.2.13 to 2.6.0.
...we updated the Sun JVM to the latest version (1.6.39) and even tried
OpenJDK.
...we increased the JVM heap size (up to 1.5gb).
...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available.
But there is good news too:
- I've just played around with the JVM's garbage collection
options. Even if the index process still hangs the JVM will no longer block.
I was able to get an interesting thread dump:
<http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
- With the addition JVM Param -XX:MaxNewSize=10m we can make the
hang occur faster (less than 100 minor gcs)
I would be very grateful if someone could have a look at this. We're running
(slowly but surely) out of ideas :-(
Thanks!
Daniel
Re: Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
One additional possibility (hardcore though) is to attach using gdb to
frozen jvm process to see where it hangs. But not sure if it will be
possible
ie. how much troubled JVM process is.
Check if it hangs always at the same place or at random places (in such case
it would be more probably HW problem (memory?)). Of course in such case you
can also use debug version of system libs or debug version of JDK/JVM to get
more details. But debug JDK is slower. But this require somebody with some
gdb / OS experience.
Marek
---------- Původní zpráva ----------
Od: Daniel Hobi <d....@gmx.ch>
Datum: 8. 3. 2013
Předmět: Re: JVM hangs during startup (indexing) (2)
"We're already monitoring the JVM process with YourKit. Looking good until
the JVM freezes, we get no more stats from the JVM.
After the freeze no more connections to the JVM could be made.
> It looks suspicious. I would recommend some HW stress test to make sure
> there is no HW problem. I am sorry I do not have better idea.
Thanks. I'll give that a try. Can you recommend a good stress test tool?
Even if the problem is not solved (yet) I want to give you a big thanks for
your support!
Daniel
Am 08.03.2013 um 11:21 schrieb mslama@email.cz:
> Last idea is to connect JVM by jconsole and monitor memory sizes if
anything
> goes wrong. (But any heap area
> overflow should not cause this. It would simply switch off something with
> warning log or throw OOME with some explanation message.)
>
> It looks suspicious. I would recommend some HW stress test to make sure
> there is no HW problem. I am sorry I do not have better idea.
>
>
>
>
>
> Marek
>
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 8. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "> Does it happen also on another machine?
>
> Yes, but only once (of 5…6 reindexing is running)
>
>> Try latest Oracle JDK to see if it makes any difference
> Same effect with latest Oracle JVM (JRE)
>
> CPU / IO looks ok to me as long as the JVM is running.
> As soon as the JVM freezes/hangs CPU/IO is idle
>
> root@miraculix:~# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 15774
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 9000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 15774
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> Thanks for your input so far!
> Daniel
>
> Am 08.03.2013 um 10:49 schrieb <ms...@email.cz>:
>
>> It is really weird that JVM process is frozen and you cannot get thread
>> dump. I never saw such problem. It looks like HW or OS problem.
>> Does it happen also on another machine? From your log I also see you use
>> OpenJDK. Try latest Oracle JDK to see if it makes any difference.
>>
>>
>>
>>
>> If you monitor frozen JVM process is there any activity? (CPU/io). Is
> there
>> any OS limit on process mem size? Check ulimit settings. (ulimit -a)
>>
>>
>>
>>
>> Marek
>>
>>
>> "Hi Marek
>>
>> I use jstack -F <pid> to get thread dumps.
>> - Except once the output of jstack is the only one I get (= the first
dump
> I
>> provided)
>> - Once (the jvm was not completely frozen, just the indexing made no
> further
>> progress) jstack also made a dump in our log file (= second dump)
>>
>> Another result after killing the jvm process with "kill -4 <pid>" (kill -
3
>
>> does not work due to a frozen jvm)
>> http://pastebin.com/qc2bUkbk
>>
>> Daniel
>>
>> Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
>>
>>> I checked 2 thread dumps you provided so I can only comment what I saw
>>> there. I did not see any deadlock. Only
>>> problem I saw was in main thread where it was waiting for connection.
>>>
>>>
>>>
>>>
>>>
>>> 1.
>>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
>>> 00007fefd3812000]
>>> 2.
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> 3.
>>> at java.lang.Object.$$YJP$$wait(Native Method)
>>> 4.
>>> at java.lang.Object.wait(Object.java)
>>> 5.
>>> at java.lang.Object.wait(Object.java:485)
>>> 6.
>>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
>>> (GenericObjectPool.java:1104)
>>> 7.
>>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
>>> GenericObjectPool$Latch)
>>> 8.
>>> at org.apache.commons.dbcp.PoolingDataSource.getConnection
>>> (PoolingDataSource.java:106)
>>> 9.
>>> at org.apache.commons.dbcp.BasicDataSource.getConnection
>>> (BasicDataSource.java:1044)
>>> 10.
>>> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
>>> getConnection(ConnectionHelper.java:439)
>>>
>>>
>>>
>>>
>>>
>>>
>>> You should be able to get thread dump any time unless JVM process is
>>> completely frozen IMO. Do you use jstack to get thread dump?
>>>
>>> Marek
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: Daniel Hobi <d....@gmx.ch>
>>> Datum: 8. 3. 2013
>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>
>>> "Hi Marek
>>>
>>> Thanks for your suggestions!
>>>
>>> Unfortunately, changing the JVM gc options so that the JVM will no
longer
>
>>> hang just worked ONCE.
>>> Therefore I was not able to get a "nice" stack trace anymore :-(
>>>
>>> Connection Pool:
>>> We had set the max connection pool size to 5. As you suggested I set it
> to
>>
>>> 15. Same effect. Even 50 or 100 did not help.
>>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is
>> this
>>> a suspicious behavior or does it simply not need more connections and is
>>> therefore not spawning new connections?
>>>
>>> Enabling debug, setting login and socket timeout on the postgres-jdbc-
>> driver
>>> did not help either.
>>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
>> core-
>>> 2.6.0 to be "sure" it is not a pool bug.
>>> Further we are not able to reproduce the hang with:
>>> - Slower machine, postgres 8.3
>>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
>>> - Faster machine, mysql
>>>
>>> This issue is slowly driving me nuts ;-)
>>>
>>> Thanks!
>>> Daniel
>>>
>>>
>>>
>>> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>>>
>>>> Also randomly we had problem with not released connections in JBoss
>>>> connection pool and Postgres. Now long time no occurrence.
>>>> In any case you can check number/list of open connections in psql . In
>> our
>>>
>>>> case it was
>>>> 97 open connection (by default 100 open connections in postgres and 3
> are
>>
>>>> reserved for admin). It happenned even if we had much smaller
>>>> max connection pool size. As it happened randomly we did not find any
>>>> solution. We tried to change connection pool size and it does not
>>>> happen any more. Still no idea what was cause of this. In any case
using
>
>>>> psql you can find out easily if it is also your problem.
>>>>
>>>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>>>> Marek
>>>>
>>>> --
>>>> Marek Slama
>>>> mslama@email.cz
>>>>
>>>>
>>>>
>>>> ---------- Původní zpráva ----------
>>>> Od: mslama@email.cz
>>>> Datum: 7. 3. 2013
>>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>>
>>>> "Quick look: It hangs in connection pool - connection pool if it cannot
>>>> return free connection to caller it waits forever by default.
>>>> How big is your connection pool? (Maximum size? It would be good to
> patch
>>
>>>> connection pool to see who/how many connections is
>>>> used and why connections are used not release. In our config we use
>>>> connection max pool size 15 and it is enough.) So first
>>>> fast fix would be to increase max connection pool size but if there is
>>>> problem with not released connection it will not help for long.
>>>>
>>>> Good news is it is not problem with memory or gc (at least IMO).
>>>>
>>>> Marek
>>>>
>>>> --
>>>> Marek Slama
>>>> mslama@email.cz
>>>>
>>>>
>>>>
>>>> ---------- Původní zpráva ----------
>>>> Od: Daniel Hobi <d....@gmx.ch>
>>>> Datum: 7. 3. 2013
>>>> Předmět: JVM hangs during startup (indexing) (2)
>>>>
>>>> "Hi there
>>>>
>>>>
>>>>
>>>> Some updates on our issue started here:
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%
3
> C
>>> 201
>>>> 30115135117.186960%40gmx.net%3E
>>>>
>>>>
>>>>
>>>> We're still facing the JVM freeze (possibly during minor gcs) even
if...
>>>>
>>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>>>
>>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>>>> OpenJDK.
>>>>
>>>> ...we increased the JVM heap size (up to 1.5gb).
>>>>
>>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x)
>> available.
>>>>
>>>>
>>>>
>>>> But there is good news too:
>>>>
>>>> - I've just played around with the JVM's garbage collection
>>>> options. Even if the index process still hangs the JVM will no longer
>>> block.
>>>> I was able to get an interesting thread dump:
>>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>>>
>>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>>>> hang occur faster (less than 100 minor gcs)
>>>>
>>>>
>>>>
>>>> I would be very grateful if someone could have a look at this. We're
>>> running
>>>> (slowly but surely) out of ideas :-(
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> Daniel"""""
>"
Re: Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
It is hard to say where problem could be. I would start with memory test (if
you use Ubuntu it has also some memory test option during
boot at grub menu. Or any similar tool.) If possible try even other machine
(3rd) which was reliable for some time.
Marek
---------- Původní zpráva ----------
Od: Daniel Hobi <d....@gmx.ch>
Datum: 8. 3. 2013
Předmět: Re: JVM hangs during startup (indexing) (2)
"We're already monitoring the JVM process with YourKit. Looking good until
the JVM freezes, we get no more stats from the JVM.
After the freeze no more connections to the JVM could be made.
> It looks suspicious. I would recommend some HW stress test to make sure
> there is no HW problem. I am sorry I do not have better idea.
Thanks. I'll give that a try. Can you recommend a good stress test tool?
Even if the problem is not solved (yet) I want to give you a big thanks for
your support!
Daniel
Am 08.03.2013 um 11:21 schrieb mslama@email.cz:
> Last idea is to connect JVM by jconsole and monitor memory sizes if
anything
> goes wrong. (But any heap area
> overflow should not cause this. It would simply switch off something with
> warning log or throw OOME with some explanation message.)
>
> It looks suspicious. I would recommend some HW stress test to make sure
> there is no HW problem. I am sorry I do not have better idea.
>
>
>
>
>
> Marek
>
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 8. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "> Does it happen also on another machine?
>
> Yes, but only once (of 5…6 reindexing is running)
>
>> Try latest Oracle JDK to see if it makes any difference
> Same effect with latest Oracle JVM (JRE)
>
> CPU / IO looks ok to me as long as the JVM is running.
> As soon as the JVM freezes/hangs CPU/IO is idle
>
> root@miraculix:~# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 15774
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 9000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 15774
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> Thanks for your input so far!
> Daniel
>
> Am 08.03.2013 um 10:49 schrieb <ms...@email.cz>:
>
>> It is really weird that JVM process is frozen and you cannot get thread
>> dump. I never saw such problem. It looks like HW or OS problem.
>> Does it happen also on another machine? From your log I also see you use
>> OpenJDK. Try latest Oracle JDK to see if it makes any difference.
>>
>>
>>
>>
>> If you monitor frozen JVM process is there any activity? (CPU/io). Is
> there
>> any OS limit on process mem size? Check ulimit settings. (ulimit -a)
>>
>>
>>
>>
>> Marek
>>
>>
>> "Hi Marek
>>
>> I use jstack -F <pid> to get thread dumps.
>> - Except once the output of jstack is the only one I get (= the first
dump
> I
>> provided)
>> - Once (the jvm was not completely frozen, just the indexing made no
> further
>> progress) jstack also made a dump in our log file (= second dump)
>>
>> Another result after killing the jvm process with "kill -4 <pid>" (kill -
3
>
>> does not work due to a frozen jvm)
>> http://pastebin.com/qc2bUkbk
>>
>> Daniel
>>
>> Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
>>
>>> I checked 2 thread dumps you provided so I can only comment what I saw
>>> there. I did not see any deadlock. Only
>>> problem I saw was in main thread where it was waiting for connection.
>>>
>>>
>>>
>>>
>>>
>>> 1.
>>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
>>> 00007fefd3812000]
>>> 2.
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> 3.
>>> at java.lang.Object.$$YJP$$wait(Native Method)
>>> 4.
>>> at java.lang.Object.wait(Object.java)
>>> 5.
>>> at java.lang.Object.wait(Object.java:485)
>>> 6.
>>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
>>> (GenericObjectPool.java:1104)
>>> 7.
>>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
>>> GenericObjectPool$Latch)
>>> 8.
>>> at org.apache.commons.dbcp.PoolingDataSource.getConnection
>>> (PoolingDataSource.java:106)
>>> 9.
>>> at org.apache.commons.dbcp.BasicDataSource.getConnection
>>> (BasicDataSource.java:1044)
>>> 10.
>>> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
>>> getConnection(ConnectionHelper.java:439)
>>>
>>>
>>>
>>>
>>>
>>>
>>> You should be able to get thread dump any time unless JVM process is
>>> completely frozen IMO. Do you use jstack to get thread dump?
>>>
>>> Marek
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: Daniel Hobi <d....@gmx.ch>
>>> Datum: 8. 3. 2013
>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>
>>> "Hi Marek
>>>
>>> Thanks for your suggestions!
>>>
>>> Unfortunately, changing the JVM gc options so that the JVM will no
longer
>
>>> hang just worked ONCE.
>>> Therefore I was not able to get a "nice" stack trace anymore :-(
>>>
>>> Connection Pool:
>>> We had set the max connection pool size to 5. As you suggested I set it
> to
>>
>>> 15. Same effect. Even 50 or 100 did not help.
>>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is
>> this
>>> a suspicious behavior or does it simply not need more connections and is
>>> therefore not spawning new connections?
>>>
>>> Enabling debug, setting login and socket timeout on the postgres-jdbc-
>> driver
>>> did not help either.
>>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
>> core-
>>> 2.6.0 to be "sure" it is not a pool bug.
>>> Further we are not able to reproduce the hang with:
>>> - Slower machine, postgres 8.3
>>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
>>> - Faster machine, mysql
>>>
>>> This issue is slowly driving me nuts ;-)
>>>
>>> Thanks!
>>> Daniel
>>>
>>>
>>>
>>> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>>>
>>>> Also randomly we had problem with not released connections in JBoss
>>>> connection pool and Postgres. Now long time no occurrence.
>>>> In any case you can check number/list of open connections in psql . In
>> our
>>>
>>>> case it was
>>>> 97 open connection (by default 100 open connections in postgres and 3
> are
>>
>>>> reserved for admin). It happenned even if we had much smaller
>>>> max connection pool size. As it happened randomly we did not find any
>>>> solution. We tried to change connection pool size and it does not
>>>> happen any more. Still no idea what was cause of this. In any case
using
>
>>>> psql you can find out easily if it is also your problem.
>>>>
>>>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>>>> Marek
>>>>
>>>> --
>>>> Marek Slama
>>>> mslama@email.cz
>>>>
>>>>
>>>>
>>>> ---------- Původní zpráva ----------
>>>> Od: mslama@email.cz
>>>> Datum: 7. 3. 2013
>>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>>
>>>> "Quick look: It hangs in connection pool - connection pool if it cannot
>>>> return free connection to caller it waits forever by default.
>>>> How big is your connection pool? (Maximum size? It would be good to
> patch
>>
>>>> connection pool to see who/how many connections is
>>>> used and why connections are used not release. In our config we use
>>>> connection max pool size 15 and it is enough.) So first
>>>> fast fix would be to increase max connection pool size but if there is
>>>> problem with not released connection it will not help for long.
>>>>
>>>> Good news is it is not problem with memory or gc (at least IMO).
>>>>
>>>> Marek
>>>>
>>>> --
>>>> Marek Slama
>>>> mslama@email.cz
>>>>
>>>>
>>>>
>>>> ---------- Původní zpráva ----------
>>>> Od: Daniel Hobi <d....@gmx.ch>
>>>> Datum: 7. 3. 2013
>>>> Předmět: JVM hangs during startup (indexing) (2)
>>>>
>>>> "Hi there
>>>>
>>>>
>>>>
>>>> Some updates on our issue started here:
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%
3
> C
>>> 201
>>>> 30115135117.186960%40gmx.net%3E
>>>>
>>>>
>>>>
>>>> We're still facing the JVM freeze (possibly during minor gcs) even
if...
>>>>
>>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>>>
>>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>>>> OpenJDK.
>>>>
>>>> ...we increased the JVM heap size (up to 1.5gb).
>>>>
>>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x)
>> available.
>>>>
>>>>
>>>>
>>>> But there is good news too:
>>>>
>>>> - I've just played around with the JVM's garbage collection
>>>> options. Even if the index process still hangs the JVM will no longer
>>> block.
>>>> I was able to get an interesting thread dump:
>>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>>>
>>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>>>> hang occur faster (less than 100 minor gcs)
>>>>
>>>>
>>>>
>>>> I would be very grateful if someone could have a look at this. We're
>>> running
>>>> (slowly but surely) out of ideas :-(
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> Daniel"""""
>"
Re: JVM hangs during startup (indexing) (2)
Posted by Daniel Hobi <d....@gmx.ch>.
We're already monitoring the JVM process with YourKit. Looking good until the JVM freezes, we get no more stats from the JVM.
After the freeze no more connections to the JVM could be made.
> It looks suspicious. I would recommend some HW stress test to make sure
> there is no HW problem. I am sorry I do not have better idea.
Thanks. I'll give that a try. Can you recommend a good stress test tool?
Even if the problem is not solved (yet) I want to give you a big thanks for your support!
Daniel
Am 08.03.2013 um 11:21 schrieb mslama@email.cz:
> Last idea is to connect JVM by jconsole and monitor memory sizes if anything
> goes wrong. (But any heap area
> overflow should not cause this. It would simply switch off something with
> warning log or throw OOME with some explanation message.)
>
> It looks suspicious. I would recommend some HW stress test to make sure
> there is no HW problem. I am sorry I do not have better idea.
>
>
>
>
>
> Marek
>
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 8. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "> Does it happen also on another machine?
>
> Yes, but only once (of 5…6 reindexing is running)
>
>> Try latest Oracle JDK to see if it makes any difference
> Same effect with latest Oracle JVM (JRE)
>
> CPU / IO looks ok to me as long as the JVM is running.
> As soon as the JVM freezes/hangs CPU/IO is idle
>
> root@miraculix:~# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 15774
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 9000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 15774
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> Thanks for your input so far!
> Daniel
>
> Am 08.03.2013 um 10:49 schrieb <ms...@email.cz>:
>
>> It is really weird that JVM process is frozen and you cannot get thread
>> dump. I never saw such problem. It looks like HW or OS problem.
>> Does it happen also on another machine? From your log I also see you use
>> OpenJDK. Try latest Oracle JDK to see if it makes any difference.
>>
>>
>>
>>
>> If you monitor frozen JVM process is there any activity? (CPU/io). Is
> there
>> any OS limit on process mem size? Check ulimit settings. (ulimit -a)
>>
>>
>>
>>
>> Marek
>>
>>
>> "Hi Marek
>>
>> I use jstack -F <pid> to get thread dumps.
>> - Except once the output of jstack is the only one I get (= the first dump
> I
>> provided)
>> - Once (the jvm was not completely frozen, just the indexing made no
> further
>> progress) jstack also made a dump in our log file (= second dump)
>>
>> Another result after killing the jvm process with "kill -4 <pid>" (kill -3
>
>> does not work due to a frozen jvm)
>> http://pastebin.com/qc2bUkbk
>>
>> Daniel
>>
>> Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
>>
>>> I checked 2 thread dumps you provided so I can only comment what I saw
>>> there. I did not see any deadlock. Only
>>> problem I saw was in main thread where it was waiting for connection.
>>>
>>>
>>>
>>>
>>>
>>> 1.
>>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
>>> 00007fefd3812000]
>>> 2.
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> 3.
>>> at java.lang.Object.$$YJP$$wait(Native Method)
>>> 4.
>>> at java.lang.Object.wait(Object.java)
>>> 5.
>>> at java.lang.Object.wait(Object.java:485)
>>> 6.
>>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
>>> (GenericObjectPool.java:1104)
>>> 7.
>>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
>>> GenericObjectPool$Latch)
>>> 8.
>>> at org.apache.commons.dbcp.PoolingDataSource.getConnection
>>> (PoolingDataSource.java:106)
>>> 9.
>>> at org.apache.commons.dbcp.BasicDataSource.getConnection
>>> (BasicDataSource.java:1044)
>>> 10.
>>> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
>>> getConnection(ConnectionHelper.java:439)
>>>
>>>
>>>
>>>
>>>
>>>
>>> You should be able to get thread dump any time unless JVM process is
>>> completely frozen IMO. Do you use jstack to get thread dump?
>>>
>>> Marek
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: Daniel Hobi <d....@gmx.ch>
>>> Datum: 8. 3. 2013
>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>
>>> "Hi Marek
>>>
>>> Thanks for your suggestions!
>>>
>>> Unfortunately, changing the JVM gc options so that the JVM will no longer
>
>>> hang just worked ONCE.
>>> Therefore I was not able to get a "nice" stack trace anymore :-(
>>>
>>> Connection Pool:
>>> We had set the max connection pool size to 5. As you suggested I set it
> to
>>
>>> 15. Same effect. Even 50 or 100 did not help.
>>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is
>> this
>>> a suspicious behavior or does it simply not need more connections and is
>>> therefore not spawning new connections?
>>>
>>> Enabling debug, setting login and socket timeout on the postgres-jdbc-
>> driver
>>> did not help either.
>>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
>> core-
>>> 2.6.0 to be "sure" it is not a pool bug.
>>> Further we are not able to reproduce the hang with:
>>> - Slower machine, postgres 8.3
>>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
>>> - Faster machine, mysql
>>>
>>> This issue is slowly driving me nuts ;-)
>>>
>>> Thanks!
>>> Daniel
>>>
>>>
>>>
>>> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>>>
>>>> Also randomly we had problem with not released connections in JBoss
>>>> connection pool and Postgres. Now long time no occurrence.
>>>> In any case you can check number/list of open connections in psql . In
>> our
>>>
>>>> case it was
>>>> 97 open connection (by default 100 open connections in postgres and 3
> are
>>
>>>> reserved for admin). It happenned even if we had much smaller
>>>> max connection pool size. As it happened randomly we did not find any
>>>> solution. We tried to change connection pool size and it does not
>>>> happen any more. Still no idea what was cause of this. In any case using
>
>>>> psql you can find out easily if it is also your problem.
>>>>
>>>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>>>> Marek
>>>>
>>>> --
>>>> Marek Slama
>>>> mslama@email.cz
>>>>
>>>>
>>>>
>>>> ---------- Původní zpráva ----------
>>>> Od: mslama@email.cz
>>>> Datum: 7. 3. 2013
>>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>>
>>>> "Quick look: It hangs in connection pool - connection pool if it cannot
>>>> return free connection to caller it waits forever by default.
>>>> How big is your connection pool? (Maximum size? It would be good to
> patch
>>
>>>> connection pool to see who/how many connections is
>>>> used and why connections are used not release. In our config we use
>>>> connection max pool size 15 and it is enough.) So first
>>>> fast fix would be to increase max connection pool size but if there is
>>>> problem with not released connection it will not help for long.
>>>>
>>>> Good news is it is not problem with memory or gc (at least IMO).
>>>>
>>>> Marek
>>>>
>>>> --
>>>> Marek Slama
>>>> mslama@email.cz
>>>>
>>>>
>>>>
>>>> ---------- Původní zpráva ----------
>>>> Od: Daniel Hobi <d....@gmx.ch>
>>>> Datum: 7. 3. 2013
>>>> Předmět: JVM hangs during startup (indexing) (2)
>>>>
>>>> "Hi there
>>>>
>>>>
>>>>
>>>> Some updates on our issue started here:
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3
> C
>>> 201
>>>> 30115135117.186960%40gmx.net%3E
>>>>
>>>>
>>>>
>>>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>>>>
>>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>>>
>>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>>>> OpenJDK.
>>>>
>>>> ...we increased the JVM heap size (up to 1.5gb).
>>>>
>>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x)
>> available.
>>>>
>>>>
>>>>
>>>> But there is good news too:
>>>>
>>>> - I've just played around with the JVM's garbage collection
>>>> options. Even if the index process still hangs the JVM will no longer
>>> block.
>>>> I was able to get an interesting thread dump:
>>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>>>
>>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>>>> hang occur faster (less than 100 minor gcs)
>>>>
>>>>
>>>>
>>>> I would be very grateful if someone could have a look at this. We're
>>> running
>>>> (slowly but surely) out of ideas :-(
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> Daniel"""""
>
Re: Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
Last idea is to connect JVM by jconsole and monitor memory sizes if anything
goes wrong. (But any heap area
overflow should not cause this. It would simply switch off something with
warning log or throw OOME with some explanation message.)
It looks suspicious. I would recommend some HW stress test to make sure
there is no HW problem. I am sorry I do not have better idea.
Marek
---------- Původní zpráva ----------
Od: Daniel Hobi <d....@gmx.ch>
Datum: 8. 3. 2013
Předmět: Re: JVM hangs during startup (indexing) (2)
"> Does it happen also on another machine?
Yes, but only once (of 5…6 reindexing is running)
> Try latest Oracle JDK to see if it makes any difference
Same effect with latest Oracle JVM (JRE)
CPU / IO looks ok to me as long as the JVM is running.
As soon as the JVM freezes/hangs CPU/IO is idle
root@miraculix:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15774
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 9000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15774
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Thanks for your input so far!
Daniel
Am 08.03.2013 um 10:49 schrieb <ms...@email.cz>:
> It is really weird that JVM process is frozen and you cannot get thread
> dump. I never saw such problem. It looks like HW or OS problem.
> Does it happen also on another machine? From your log I also see you use
> OpenJDK. Try latest Oracle JDK to see if it makes any difference.
>
>
>
>
> If you monitor frozen JVM process is there any activity? (CPU/io). Is
there
> any OS limit on process mem size? Check ulimit settings. (ulimit -a)
>
>
>
>
> Marek
>
>
> "Hi Marek
>
> I use jstack -F <pid> to get thread dumps.
> - Except once the output of jstack is the only one I get (= the first dump
I
> provided)
> - Once (the jvm was not completely frozen, just the indexing made no
further
> progress) jstack also made a dump in our log file (= second dump)
>
> Another result after killing the jvm process with "kill -4 <pid>" (kill -3
> does not work due to a frozen jvm)
> http://pastebin.com/qc2bUkbk
>
> Daniel
>
> Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
>
>> I checked 2 thread dumps you provided so I can only comment what I saw
>> there. I did not see any deadlock. Only
>> problem I saw was in main thread where it was waiting for connection.
>>
>>
>>
>>
>>
>> 1.
>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
>> 00007fefd3812000]
>> 2.
>> java.lang.Thread.State: WAITING (on object monitor)
>> 3.
>> at java.lang.Object.$$YJP$$wait(Native Method)
>> 4.
>> at java.lang.Object.wait(Object.java)
>> 5.
>> at java.lang.Object.wait(Object.java:485)
>> 6.
>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
>> (GenericObjectPool.java:1104)
>> 7.
>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
>> GenericObjectPool$Latch)
>> 8.
>> at org.apache.commons.dbcp.PoolingDataSource.getConnection
>> (PoolingDataSource.java:106)
>> 9.
>> at org.apache.commons.dbcp.BasicDataSource.getConnection
>> (BasicDataSource.java:1044)
>> 10.
>> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
>> getConnection(ConnectionHelper.java:439)
>>
>>
>>
>>
>>
>>
>> You should be able to get thread dump any time unless JVM process is
>> completely frozen IMO. Do you use jstack to get thread dump?
>>
>> Marek
>>
>>
>> ---------- Původní zpráva ----------
>> Od: Daniel Hobi <d....@gmx.ch>
>> Datum: 8. 3. 2013
>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>
>> "Hi Marek
>>
>> Thanks for your suggestions!
>>
>> Unfortunately, changing the JVM gc options so that the JVM will no longer
>> hang just worked ONCE.
>> Therefore I was not able to get a "nice" stack trace anymore :-(
>>
>> Connection Pool:
>> We had set the max connection pool size to 5. As you suggested I set it
to
>
>> 15. Same effect. Even 50 or 100 did not help.
>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is
> this
>> a suspicious behavior or does it simply not need more connections and is
>> therefore not spawning new connections?
>>
>> Enabling debug, setting login and socket timeout on the postgres-jdbc-
> driver
>> did not help either.
>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
> core-
>> 2.6.0 to be "sure" it is not a pool bug.
>> Further we are not able to reproduce the hang with:
>> - Slower machine, postgres 8.3
>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
>> - Faster machine, mysql
>>
>> This issue is slowly driving me nuts ;-)
>>
>> Thanks!
>> Daniel
>>
>>
>>
>> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>>
>>> Also randomly we had problem with not released connections in JBoss
>>> connection pool and Postgres. Now long time no occurrence.
>>> In any case you can check number/list of open connections in psql . In
> our
>>
>>> case it was
>>> 97 open connection (by default 100 open connections in postgres and 3
are
>
>>> reserved for admin). It happenned even if we had much smaller
>>> max connection pool size. As it happened randomly we did not find any
>>> solution. We tried to change connection pool size and it does not
>>> happen any more. Still no idea what was cause of this. In any case using
>>> psql you can find out easily if it is also your problem.
>>>
>>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>>> Marek
>>>
>>> --
>>> Marek Slama
>>> mslama@email.cz
>>>
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: mslama@email.cz
>>> Datum: 7. 3. 2013
>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>
>>> "Quick look: It hangs in connection pool - connection pool if it cannot
>>> return free connection to caller it waits forever by default.
>>> How big is your connection pool? (Maximum size? It would be good to
patch
>
>>> connection pool to see who/how many connections is
>>> used and why connections are used not release. In our config we use
>>> connection max pool size 15 and it is enough.) So first
>>> fast fix would be to increase max connection pool size but if there is
>>> problem with not released connection it will not help for long.
>>>
>>> Good news is it is not problem with memory or gc (at least IMO).
>>>
>>> Marek
>>>
>>> --
>>> Marek Slama
>>> mslama@email.cz
>>>
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: Daniel Hobi <d....@gmx.ch>
>>> Datum: 7. 3. 2013
>>> Předmět: JVM hangs during startup (indexing) (2)
>>>
>>> "Hi there
>>>
>>>
>>>
>>> Some updates on our issue started here:
>>>
>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3
C
>> 201
>>> 30115135117.186960%40gmx.net%3E
>>>
>>>
>>>
>>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>>>
>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>>
>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>>> OpenJDK.
>>>
>>> ...we increased the JVM heap size (up to 1.5gb).
>>>
>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x)
> available.
>>>
>>>
>>>
>>> But there is good news too:
>>>
>>> - I've just played around with the JVM's garbage collection
>>> options. Even if the index process still hangs the JVM will no longer
>> block.
>>> I was able to get an interesting thread dump:
>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>>
>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>>> hang occur faster (less than 100 minor gcs)
>>>
>>>
>>>
>>> I would be very grateful if someone could have a look at this. We're
>> running
>>> (slowly but surely) out of ideas :-(
>>>
>>>
>>>
>>> Thanks!
>>>
>>> Daniel"""""
Re: JVM hangs during startup (indexing) (2)
Posted by Daniel Hobi <d....@gmx.ch>.
> Does it happen also on another machine?
Yes, but only once (of 5…6 reindexing is running)
> Try latest Oracle JDK to see if it makes any difference
Same effect with latest Oracle JVM (JRE)
CPU / IO looks ok to me as long as the JVM is running.
As soon as the JVM freezes/hangs CPU/IO is idle
root@miraculix:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15774
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 9000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15774
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Thanks for your input so far!
Daniel
Am 08.03.2013 um 10:49 schrieb <ms...@email.cz>:
> It is really weird that JVM process is frozen and you cannot get thread
> dump. I never saw such problem. It looks like HW or OS problem.
> Does it happen also on another machine? From your log I also see you use
> OpenJDK. Try latest Oracle JDK to see if it makes any difference.
>
>
>
>
> If you monitor frozen JVM process is there any activity? (CPU/io). Is there
> any OS limit on process mem size? Check ulimit settings. (ulimit -a)
>
>
>
>
> Marek
>
>
> "Hi Marek
>
> I use jstack -F <pid> to get thread dumps.
> - Except once the output of jstack is the only one I get (= the first dump I
> provided)
> - Once (the jvm was not completely frozen, just the indexing made no further
> progress) jstack also made a dump in our log file (= second dump)
>
> Another result after killing the jvm process with "kill -4 <pid>" (kill -3
> does not work due to a frozen jvm)
> http://pastebin.com/qc2bUkbk
>
> Daniel
>
> Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
>
>> I checked 2 thread dumps you provided so I can only comment what I saw
>> there. I did not see any deadlock. Only
>> problem I saw was in main thread where it was waiting for connection.
>>
>>
>>
>>
>>
>> 1.
>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
>> 00007fefd3812000]
>> 2.
>> java.lang.Thread.State: WAITING (on object monitor)
>> 3.
>> at java.lang.Object.$$YJP$$wait(Native Method)
>> 4.
>> at java.lang.Object.wait(Object.java)
>> 5.
>> at java.lang.Object.wait(Object.java:485)
>> 6.
>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
>> (GenericObjectPool.java:1104)
>> 7.
>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
>> GenericObjectPool$Latch)
>> 8.
>> at org.apache.commons.dbcp.PoolingDataSource.getConnection
>> (PoolingDataSource.java:106)
>> 9.
>> at org.apache.commons.dbcp.BasicDataSource.getConnection
>> (BasicDataSource.java:1044)
>> 10.
>> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
>> getConnection(ConnectionHelper.java:439)
>>
>>
>>
>>
>>
>>
>> You should be able to get thread dump any time unless JVM process is
>> completely frozen IMO. Do you use jstack to get thread dump?
>>
>> Marek
>>
>>
>> ---------- Původní zpráva ----------
>> Od: Daniel Hobi <d....@gmx.ch>
>> Datum: 8. 3. 2013
>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>
>> "Hi Marek
>>
>> Thanks for your suggestions!
>>
>> Unfortunately, changing the JVM gc options so that the JVM will no longer
>> hang just worked ONCE.
>> Therefore I was not able to get a "nice" stack trace anymore :-(
>>
>> Connection Pool:
>> We had set the max connection pool size to 5. As you suggested I set it to
>
>> 15. Same effect. Even 50 or 100 did not help.
>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is
> this
>> a suspicious behavior or does it simply not need more connections and is
>> therefore not spawning new connections?
>>
>> Enabling debug, setting login and socket timeout on the postgres-jdbc-
> driver
>> did not help either.
>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
> core-
>> 2.6.0 to be "sure" it is not a pool bug.
>> Further we are not able to reproduce the hang with:
>> - Slower machine, postgres 8.3
>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
>> - Faster machine, mysql
>>
>> This issue is slowly driving me nuts ;-)
>>
>> Thanks!
>> Daniel
>>
>>
>>
>> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>>
>>> Also randomly we had problem with not released connections in JBoss
>>> connection pool and Postgres. Now long time no occurrence.
>>> In any case you can check number/list of open connections in psql . In
> our
>>
>>> case it was
>>> 97 open connection (by default 100 open connections in postgres and 3 are
>
>>> reserved for admin). It happenned even if we had much smaller
>>> max connection pool size. As it happened randomly we did not find any
>>> solution. We tried to change connection pool size and it does not
>>> happen any more. Still no idea what was cause of this. In any case using
>>> psql you can find out easily if it is also your problem.
>>>
>>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>>> Marek
>>>
>>> --
>>> Marek Slama
>>> mslama@email.cz
>>>
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: mslama@email.cz
>>> Datum: 7. 3. 2013
>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>>
>>> "Quick look: It hangs in connection pool - connection pool if it cannot
>>> return free connection to caller it waits forever by default.
>>> How big is your connection pool? (Maximum size? It would be good to patch
>
>>> connection pool to see who/how many connections is
>>> used and why connections are used not release. In our config we use
>>> connection max pool size 15 and it is enough.) So first
>>> fast fix would be to increase max connection pool size but if there is
>>> problem with not released connection it will not help for long.
>>>
>>> Good news is it is not problem with memory or gc (at least IMO).
>>>
>>> Marek
>>>
>>> --
>>> Marek Slama
>>> mslama@email.cz
>>>
>>>
>>>
>>> ---------- Původní zpráva ----------
>>> Od: Daniel Hobi <d....@gmx.ch>
>>> Datum: 7. 3. 2013
>>> Předmět: JVM hangs during startup (indexing) (2)
>>>
>>> "Hi there
>>>
>>>
>>>
>>> Some updates on our issue started here:
>>>
>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C
>> 201
>>> 30115135117.186960%40gmx.net%3E
>>>
>>>
>>>
>>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>>>
>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>>
>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>>> OpenJDK.
>>>
>>> ...we increased the JVM heap size (up to 1.5gb).
>>>
>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x)
> available.
>>>
>>>
>>>
>>> But there is good news too:
>>>
>>> - I've just played around with the JVM's garbage collection
>>> options. Even if the index process still hangs the JVM will no longer
>> block.
>>> I was able to get an interesting thread dump:
>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>>
>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>>> hang occur faster (less than 100 minor gcs)
>>>
>>>
>>>
>>> I would be very grateful if someone could have a look at this. We're
>> running
>>> (slowly but surely) out of ideas :-(
>>>
>>>
>>>
>>> Thanks!
>>>
>>> Daniel""""
Re: Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
It is really weird that JVM process is frozen and you cannot get thread
dump. I never saw such problem. It looks like HW or OS problem.
Does it happen also on another machine? From your log I also see you use
OpenJDK. Try latest Oracle JDK to see if it makes any difference.
If you monitor frozen JVM process is there any activity? (CPU/io). Is there
any OS limit on process mem size? Check ulimit settings. (ulimit -a)
Marek
"Hi Marek
I use jstack -F <pid> to get thread dumps.
- Except once the output of jstack is the only one I get (= the first dump I
provided)
- Once (the jvm was not completely frozen, just the indexing made no further
progress) jstack also made a dump in our log file (= second dump)
Another result after killing the jvm process with "kill -4 <pid>" (kill -3
does not work due to a frozen jvm)
http://pastebin.com/qc2bUkbk
Daniel
Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
> I checked 2 thread dumps you provided so I can only comment what I saw
> there. I did not see any deadlock. Only
> problem I saw was in main thread where it was waiting for connection.
>
>
>
>
>
> 1.
> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
> 00007fefd3812000]
> 2.
> java.lang.Thread.State: WAITING (on object monitor)
> 3.
> at java.lang.Object.$$YJP$$wait(Native Method)
> 4.
> at java.lang.Object.wait(Object.java)
> 5.
> at java.lang.Object.wait(Object.java:485)
> 6.
> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
> (GenericObjectPool.java:1104)
> 7.
> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
> GenericObjectPool$Latch)
> 8.
> at org.apache.commons.dbcp.PoolingDataSource.getConnection
> (PoolingDataSource.java:106)
> 9.
> at org.apache.commons.dbcp.BasicDataSource.getConnection
> (BasicDataSource.java:1044)
> 10.
> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
> getConnection(ConnectionHelper.java:439)
>
>
>
>
>
>
> You should be able to get thread dump any time unless JVM process is
> completely frozen IMO. Do you use jstack to get thread dump?
>
> Marek
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 8. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "Hi Marek
>
> Thanks for your suggestions!
>
> Unfortunately, changing the JVM gc options so that the JVM will no longer
> hang just worked ONCE.
> Therefore I was not able to get a "nice" stack trace anymore :-(
>
> Connection Pool:
> We had set the max connection pool size to 5. As you suggested I set it to
> 15. Same effect. Even 50 or 100 did not help.
> Interestingly, postgres shows just 3 open connection for jackrabbit. Is
this
> a suspicious behavior or does it simply not need more connections and is
> therefore not spawning new connections?
>
> Enabling debug, setting login and socket timeout on the postgres-jdbc-
driver
> did not help either.
> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
core-
> 2.6.0 to be "sure" it is not a pool bug.
> Further we are not able to reproduce the hang with:
> - Slower machine, postgres 8.3
> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
> - Faster machine, mysql
>
> This issue is slowly driving me nuts ;-)
>
> Thanks!
> Daniel
>
>
>
> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>
>> Also randomly we had problem with not released connections in JBoss
>> connection pool and Postgres. Now long time no occurrence.
>> In any case you can check number/list of open connections in psql . In
our
>
>> case it was
>> 97 open connection (by default 100 open connections in postgres and 3 are
>> reserved for admin). It happenned even if we had much smaller
>> max connection pool size. As it happened randomly we did not find any
>> solution. We tried to change connection pool size and it does not
>> happen any more. Still no idea what was cause of this. In any case using
>> psql you can find out easily if it is also your problem.
>>
>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>> Marek
>>
>> --
>> Marek Slama
>> mslama@email.cz
>>
>>
>>
>> ---------- Původní zpráva ----------
>> Od: mslama@email.cz
>> Datum: 7. 3. 2013
>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>
>> "Quick look: It hangs in connection pool - connection pool if it cannot
>> return free connection to caller it waits forever by default.
>> How big is your connection pool? (Maximum size? It would be good to patch
>> connection pool to see who/how many connections is
>> used and why connections are used not release. In our config we use
>> connection max pool size 15 and it is enough.) So first
>> fast fix would be to increase max connection pool size but if there is
>> problem with not released connection it will not help for long.
>>
>> Good news is it is not problem with memory or gc (at least IMO).
>>
>> Marek
>>
>> --
>> Marek Slama
>> mslama@email.cz
>>
>>
>>
>> ---------- Původní zpráva ----------
>> Od: Daniel Hobi <d....@gmx.ch>
>> Datum: 7. 3. 2013
>> Předmět: JVM hangs during startup (indexing) (2)
>>
>> "Hi there
>>
>>
>>
>> Some updates on our issue started here:
>>
>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C
> 201
>> 30115135117.186960%40gmx.net%3E
>>
>>
>>
>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>>
>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>
>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>> OpenJDK.
>>
>> ...we increased the JVM heap size (up to 1.5gb).
>>
>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x)
available.
>>
>>
>>
>> But there is good news too:
>>
>> - I've just played around with the JVM's garbage collection
>> options. Even if the index process still hangs the JVM will no longer
> block.
>> I was able to get an interesting thread dump:
>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>
>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>> hang occur faster (less than 100 minor gcs)
>>
>>
>>
>> I would be very grateful if someone could have a look at this. We're
> running
>> (slowly but surely) out of ideas :-(
>>
>>
>>
>> Thanks!
>>
>> Daniel""""
Re: JVM hangs during startup (indexing) (2)
Posted by Daniel Hobi <d....@gmx.ch>.
Hi Marek
I use jstack -F <pid> to get thread dumps.
- Except once the output of jstack is the only one I get (= the first dump I provided)
- Once (the jvm was not completely frozen, just the indexing made no further progress) jstack also made a dump in our log file (= second dump)
Another result after killing the jvm process with "kill -4 <pid>" (kill -3 does not work due to a frozen jvm)
http://pastebin.com/qc2bUkbk
Daniel
Am 08.03.2013 um 10:16 schrieb <ms...@email.cz>:
> I checked 2 thread dumps you provided so I can only comment what I saw
> there. I did not see any deadlock. Only
> problem I saw was in main thread where it was waiting for connection.
>
>
>
>
>
> 1.
> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
> 00007fefd3812000]
> 2.
> java.lang.Thread.State: WAITING (on object monitor)
> 3.
> at java.lang.Object.$$YJP$$wait(Native Method)
> 4.
> at java.lang.Object.wait(Object.java)
> 5.
> at java.lang.Object.wait(Object.java:485)
> 6.
> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
> (GenericObjectPool.java:1104)
> 7.
> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
> GenericObjectPool$Latch)
> 8.
> at org.apache.commons.dbcp.PoolingDataSource.getConnection
> (PoolingDataSource.java:106)
> 9.
> at org.apache.commons.dbcp.BasicDataSource.getConnection
> (BasicDataSource.java:1044)
> 10.
> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
> getConnection(ConnectionHelper.java:439)
>
>
>
>
>
>
> You should be able to get thread dump any time unless JVM process is
> completely frozen IMO. Do you use jstack to get thread dump?
>
> Marek
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 8. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "Hi Marek
>
> Thanks for your suggestions!
>
> Unfortunately, changing the JVM gc options so that the JVM will no longer
> hang just worked ONCE.
> Therefore I was not able to get a "nice" stack trace anymore :-(
>
> Connection Pool:
> We had set the max connection pool size to 5. As you suggested I set it to
> 15. Same effect. Even 50 or 100 did not help.
> Interestingly, postgres shows just 3 open connection for jackrabbit. Is this
> a suspicious behavior or does it simply not need more connections and is
> therefore not spawning new connections?
>
> Enabling debug, setting login and socket timeout on the postgres-jdbc-driver
> did not help either.
> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-core-
> 2.6.0 to be "sure" it is not a pool bug.
> Further we are not able to reproduce the hang with:
> - Slower machine, postgres 8.3
> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
> - Faster machine, mysql
>
> This issue is slowly driving me nuts ;-)
>
> Thanks!
> Daniel
>
>
>
> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>
>> Also randomly we had problem with not released connections in JBoss
>> connection pool and Postgres. Now long time no occurrence.
>> In any case you can check number/list of open connections in psql . In our
>
>> case it was
>> 97 open connection (by default 100 open connections in postgres and 3 are
>> reserved for admin). It happenned even if we had much smaller
>> max connection pool size. As it happened randomly we did not find any
>> solution. We tried to change connection pool size and it does not
>> happen any more. Still no idea what was cause of this. In any case using
>> psql you can find out easily if it is also your problem.
>>
>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>> Marek
>>
>> --
>> Marek Slama
>> mslama@email.cz
>>
>>
>>
>> ---------- Původní zpráva ----------
>> Od: mslama@email.cz
>> Datum: 7. 3. 2013
>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>
>> "Quick look: It hangs in connection pool - connection pool if it cannot
>> return free connection to caller it waits forever by default.
>> How big is your connection pool? (Maximum size? It would be good to patch
>> connection pool to see who/how many connections is
>> used and why connections are used not release. In our config we use
>> connection max pool size 15 and it is enough.) So first
>> fast fix would be to increase max connection pool size but if there is
>> problem with not released connection it will not help for long.
>>
>> Good news is it is not problem with memory or gc (at least IMO).
>>
>> Marek
>>
>> --
>> Marek Slama
>> mslama@email.cz
>>
>>
>>
>> ---------- Původní zpráva ----------
>> Od: Daniel Hobi <d....@gmx.ch>
>> Datum: 7. 3. 2013
>> Předmět: JVM hangs during startup (indexing) (2)
>>
>> "Hi there
>>
>>
>>
>> Some updates on our issue started here:
>>
>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C
> 201
>> 30115135117.186960%40gmx.net%3E
>>
>>
>>
>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>>
>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>
>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>> OpenJDK.
>>
>> ...we increased the JVM heap size (up to 1.5gb).
>>
>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available.
>>
>>
>>
>> But there is good news too:
>>
>> - I've just played around with the JVM's garbage collection
>> options. Even if the index process still hangs the JVM will no longer
> block.
>> I was able to get an interesting thread dump:
>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>
>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>> hang occur faster (less than 100 minor gcs)
>>
>>
>>
>> I would be very grateful if someone could have a look at this. We're
> running
>> (slowly but surely) out of ideas :-(
>>
>>
>>
>> Thanks!
>>
>> Daniel"""
Re: Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
I checked 2 thread dumps you provided so I can only comment what I saw
there. I did not see any deadlock. Only
problem I saw was in main thread where it was waiting for connection.
1.
"main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
00007fefd3812000]
2.
java.lang.Thread.State: WAITING (on object monitor)
3.
at java.lang.Object.$$YJP$$wait(Native Method)
4.
at java.lang.Object.wait(Object.java)
5.
at java.lang.Object.wait(Object.java:485)
6.
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
(GenericObjectPool.java:1104)
7.
- locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
GenericObjectPool$Latch)
8.
at org.apache.commons.dbcp.PoolingDataSource.getConnection
(PoolingDataSource.java:106)
9.
at org.apache.commons.dbcp.BasicDataSource.getConnection
(BasicDataSource.java:1044)
10.
at org.apache.jackrabbit.core.util.db.ConnectionHelper.
getConnection(ConnectionHelper.java:439)
You should be able to get thread dump any time unless JVM process is
completely frozen IMO. Do you use jstack to get thread dump?
Marek
---------- Původní zpráva ----------
Od: Daniel Hobi <d....@gmx.ch>
Datum: 8. 3. 2013
Předmět: Re: JVM hangs during startup (indexing) (2)
"Hi Marek
Thanks for your suggestions!
Unfortunately, changing the JVM gc options so that the JVM will no longer
hang just worked ONCE.
Therefore I was not able to get a "nice" stack trace anymore :-(
Connection Pool:
We had set the max connection pool size to 5. As you suggested I set it to
15. Same effect. Even 50 or 100 did not help.
Interestingly, postgres shows just 3 open connection for jackrabbit. Is this
a suspicious behavior or does it simply not need more connections and is
therefore not spawning new connections?
Enabling debug, setting login and socket timeout on the postgres-jdbc-driver
did not help either.
I even changed the pool implementation from dbcp to c3p0 in jackrabbit-core-
2.6.0 to be "sure" it is not a pool bug.
Further we are not able to reproduce the hang with:
- Slower machine, postgres 8.3
- Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
- Faster machine, mysql
This issue is slowly driving me nuts ;-)
Thanks!
Daniel
Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
> Also randomly we had problem with not released connections in JBoss
> connection pool and Postgres. Now long time no occurrence.
> In any case you can check number/list of open connections in psql . In our
> case it was
> 97 open connection (by default 100 open connections in postgres and 3 are
> reserved for admin). It happenned even if we had much smaller
> max connection pool size. As it happened randomly we did not find any
> solution. We tried to change connection pool size and it does not
> happen any more. Still no idea what was cause of this. In any case using
> psql you can find out easily if it is also your problem.
>
> Query is for 9.x: SELECT * FROM pg_stat_activity;
> Marek
>
> --
> Marek Slama
> mslama@email.cz
>
>
>
> ---------- Původní zpráva ----------
> Od: mslama@email.cz
> Datum: 7. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "Quick look: It hangs in connection pool - connection pool if it cannot
> return free connection to caller it waits forever by default.
> How big is your connection pool? (Maximum size? It would be good to patch
> connection pool to see who/how many connections is
> used and why connections are used not release. In our config we use
> connection max pool size 15 and it is enough.) So first
> fast fix would be to increase max connection pool size but if there is
> problem with not released connection it will not help for long.
>
> Good news is it is not problem with memory or gc (at least IMO).
>
> Marek
>
> --
> Marek Slama
> mslama@email.cz
>
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 7. 3. 2013
> Předmět: JVM hangs during startup (indexing) (2)
>
> "Hi there
>
>
>
> Some updates on our issue started here:
>
> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C
201
> 30115135117.186960%40gmx.net%3E
>
>
>
> We're still facing the JVM freeze (possibly during minor gcs) even if...
>
> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>
> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
> OpenJDK.
>
> ...we increased the JVM heap size (up to 1.5gb).
>
> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available.
>
>
>
> But there is good news too:
>
> - I've just played around with the JVM's garbage collection
> options. Even if the index process still hangs the JVM will no longer
block.
> I was able to get an interesting thread dump:
> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>
> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
> hang occur faster (less than 100 minor gcs)
>
>
>
> I would be very grateful if someone could have a look at this. We're
running
> (slowly but surely) out of ideas :-(
>
>
>
> Thanks!
>
> Daniel"""
Re: JVM hangs during startup (indexing) (2)
Posted by Daniel Hobi <d....@gmx.ch>.
Hi Marek
Thanks for your suggestions!
Unfortunately, changing the JVM gc options so that the JVM will no longer hang just worked ONCE.
Therefore I was not able to get a "nice" stack trace anymore :-(
Connection Pool:
We had set the max connection pool size to 5. As you suggested I set it to 15. Same effect. Even 50 or 100 did not help.
Interestingly, postgres shows just 3 open connection for jackrabbit. Is this a suspicious behavior or does it simply not need more connections and is therefore not spawning new connections?
Enabling debug, setting login and socket timeout on the postgres-jdbc-driver did not help either.
I even changed the pool implementation from dbcp to c3p0 in jackrabbit-core-2.6.0 to be "sure" it is not a pool bug.
Further we are not able to reproduce the hang with:
- Slower machine, postgres 8.3
- Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
- Faster machine, mysql
This issue is slowly driving me nuts ;-)
Thanks!
Daniel
Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
> Also randomly we had problem with not released connections in JBoss
> connection pool and Postgres. Now long time no occurrence.
> In any case you can check number/list of open connections in psql . In our
> case it was
> 97 open connection (by default 100 open connections in postgres and 3 are
> reserved for admin). It happenned even if we had much smaller
> max connection pool size. As it happened randomly we did not find any
> solution. We tried to change connection pool size and it does not
> happen any more. Still no idea what was cause of this. In any case using
> psql you can find out easily if it is also your problem.
>
> Query is for 9.x: SELECT * FROM pg_stat_activity;
> Marek
>
> --
> Marek Slama
> mslama@email.cz
>
>
>
> ---------- Původní zpráva ----------
> Od: mslama@email.cz
> Datum: 7. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
>
> "Quick look: It hangs in connection pool - connection pool if it cannot
> return free connection to caller it waits forever by default.
> How big is your connection pool? (Maximum size? It would be good to patch
> connection pool to see who/how many connections is
> used and why connections are used not release. In our config we use
> connection max pool size 15 and it is enough.) So first
> fast fix would be to increase max connection pool size but if there is
> problem with not released connection it will not help for long.
>
> Good news is it is not problem with memory or gc (at least IMO).
>
> Marek
>
> --
> Marek Slama
> mslama@email.cz
>
>
>
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d....@gmx.ch>
> Datum: 7. 3. 2013
> Předmět: JVM hangs during startup (indexing) (2)
>
> "Hi there
>
>
>
> Some updates on our issue started here:
>
> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C201
> 30115135117.186960%40gmx.net%3E
>
>
>
> We're still facing the JVM freeze (possibly during minor gcs) even if...
>
> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>
> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
> OpenJDK.
>
> ...we increased the JVM heap size (up to 1.5gb).
>
> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available.
>
>
>
> But there is good news too:
>
> - I've just played around with the JVM's garbage collection
> options. Even if the index process still hangs the JVM will no longer block.
> I was able to get an interesting thread dump:
> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>
> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
> hang occur faster (less than 100 minor gcs)
>
>
>
> I would be very grateful if someone could have a look at this. We're running
> (slowly but surely) out of ideas :-(
>
>
>
> Thanks!
>
> Daniel""
Re: Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
Also randomly we had problem with not released connections in JBoss
connection pool and Postgres. Now long time no occurrence.
In any case you can check number/list of open connections in psql . In our
case it was
97 open connection (by default 100 open connections in postgres and 3 are
reserved for admin). It happenned even if we had much smaller
max connection pool size. As it happened randomly we did not find any
solution. We tried to change connection pool size and it does not
happen any more. Still no idea what was cause of this. In any case using
psql you can find out easily if it is also your problem.
Query is for 9.x: SELECT * FROM pg_stat_activity;
Marek
--
Marek Slama
mslama@email.cz
---------- Původní zpráva ----------
Od: mslama@email.cz
Datum: 7. 3. 2013
Předmět: Re: JVM hangs during startup (indexing) (2)
"Quick look: It hangs in connection pool - connection pool if it cannot
return free connection to caller it waits forever by default.
How big is your connection pool? (Maximum size? It would be good to patch
connection pool to see who/how many connections is
used and why connections are used not release. In our config we use
connection max pool size 15 and it is enough.) So first
fast fix would be to increase max connection pool size but if there is
problem with not released connection it will not help for long.
Good news is it is not problem with memory or gc (at least IMO).
Marek
--
Marek Slama
mslama@email.cz
---------- Původní zpráva ----------
Od: Daniel Hobi <d....@gmx.ch>
Datum: 7. 3. 2013
Předmět: JVM hangs during startup (indexing) (2)
"Hi there
Some updates on our issue started here:
http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C201
30115135117.186960%40gmx.net%3E
We're still facing the JVM freeze (possibly during minor gcs) even if...
...we updated Jackrabbit from 2.2.13 to 2.6.0.
...we updated the Sun JVM to the latest version (1.6.39) and even tried
OpenJDK.
...we increased the JVM heap size (up to 1.5gb).
...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available.
But there is good news too:
- I've just played around with the JVM's garbage collection
options. Even if the index process still hangs the JVM will no longer block.
I was able to get an interesting thread dump:
<http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
- With the addition JVM Param -XX:MaxNewSize=10m we can make the
hang occur faster (less than 100 minor gcs)
I would be very grateful if someone could have a look at this. We're running
(slowly but surely) out of ideas :-(
Thanks!
Daniel""
Re: JVM hangs during startup (indexing) (2)
Posted by ms...@email.cz.
Quick look: It hangs in connection pool - connection pool if it cannot
return free connection to caller it waits forever by default.
How big is your connection pool? (Maximum size? It would be good to patch
connection pool to see who/how many connections is
used and why connections are used not release. In our config we use
connection max pool size 15 and it is enough.) So first
fast fix would be to increase max connection pool size but if there is
problem with not released connection it will not help for long.
Good news is it is not problem with memory or gc (at least IMO).
Marek
--
Marek Slama
mslama@email.cz
---------- Původní zpráva ----------
Od: Daniel Hobi <d....@gmx.ch>
Datum: 7. 3. 2013
Předmět: JVM hangs during startup (indexing) (2)
"Hi there
Some updates on our issue started here:
http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C201
30115135117.186960%40gmx.net%3E
We're still facing the JVM freeze (possibly during minor gcs) even if...
...we updated Jackrabbit from 2.2.13 to 2.6.0.
...we updated the Sun JVM to the latest version (1.6.39) and even tried
OpenJDK.
...we increased the JVM heap size (up to 1.5gb).
...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available.
But there is good news too:
- I've just played around with the JVM's garbage collection
options. Even if the index process still hangs the JVM will no longer block.
I was able to get an interesting thread dump:
<http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
- With the addition JVM Param -XX:MaxNewSize=10m we can make the
hang occur faster (less than 100 minor gcs)
I would be very grateful if someone could have a look at this. We're running
(slowly but surely) out of ideas :-(
Thanks!
Daniel"
Re: JVM hangs during startup (indexing) (2)
Posted by Hendry_Betts <he...@greenskycredit.com>.
Daniel,
We had a similar issue in our environment as our JCR contained many XML
documents. It would hang during indexing and we were able to trace it to the
Lucene components. The fix, albeit sub-optimal, was to change how indexing
behaved through the use of an indexing_configuration.xml that reduced the
memory load and sped up indexing significantly.
We also turned off index highlighting by setting the following node inside
the SearchIndex stanza and the SearchIndex child node of the Workspace
stanza.
It seemed counter-intuitive at first, that this should make a difference,
but it does.
HTH
--
View this message in context: http://jackrabbit.510166.n4.nabble.com/JVM-hangs-during-startup-indexing-2-tp4657888p4657916.html
Sent from the Jackrabbit - Users mailing list archive at Nabble.com.