DO NOT REPLY [Bug 46218] New: deadlock on xindice+jetty shutdown

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46218

           Summary: deadlock on xindice+jetty shutdown
           Product: Xindice
           Version: unspecified
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: critical
          Priority: P2
         Component: Server
        AssignedTo: xindice-dev@xml.apache.org
        ReportedBy: mike@vee.net


When running Xindice using the shipped copy of Jetty (and probably any kind of
servlet container that installs its own shutdown hook), a race exists on
shutdown which can lead to deadlock.

Using the xindice script to stop a running instances sends a TERM to the JVM,
thus causing all hooks registered with Runtime.addShutdownHook() to be
executed. Both Xindice and Jetty register hooks - DatabaseShutdownHandler and
in org.mortbay.jetty.Server respectively.

Deadlock occurs in at least this situation: Jetty's shutdown hook thread starts
running, calls XindiceServlet.destroy(), which calls Database.close(boolean) on
an instance, synchronizing on that instance. DatabaseShutdownHandler's thread
starts running and synchronizes on its set of databases. Jetty's thread in
Database.close(boolean) calls DatabaseShutdownHandler.removeDatabase(), tries
to also synchronize on the handler's set of instances but is blocked, since the
handler has already done so. The handler tries to synchronize on the Database
instance, but is blocked since the Jetty thread has already done so. Bam!
Deadlock.

There may be other orderings of operations that exhibit the same problem.

I'm seeing this once in every 10-20 shutdowns on a RHEL 5 box w/ Java 1.6.0_07.

This can be worked-around by disabling Jetty's shutdown hook, for example, by
passing "-DJETTY_NO_SHUTDOWN_HOOK=true" as an argument to the JVM on startup,
but that could be problematic if any requests are in progress on shutdown.

Disabling Xindice's handler would probably be the correct thing to do when
running embedded in a servlet container that reliably manages its own
lifecycle, but this does not appear possible without with code changes.

A fix might be to add a synchronized shutdown() method somewhere that both
XindiceServlet and DatabaseShutdownHandler can call, thus ensuring that only
one can enter it at a time, and obviating the problem of both trying to do so
at the same time.


-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

DO NOT REPLY [Bug 46218] deadlock on xindice+jetty shutdown

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46218


Vadim Gritsenko <vg...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED




--- Comment #2 from Vadim Gritsenko <vg...@apache.org>  2008-11-15 20:43:06 PST ---
Thanks for the detailed bug report. I think it might be better just to change
synchronization sequence in Database.close(); so i went ahead and changed it in
r717978 ( https://svn.apache.org/viewcvs.cgi?view=rev&rev=717978 ) on trunk. Would you mind testing it on your system? Thanks.


-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

DO NOT REPLY [Bug 46218] deadlock on xindice+jetty shutdown

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46218





--- Comment #1 from Michael Gratton <mi...@vee.net>  2008-11-15 18:45:03 PST ---
I should note that this is can cause data loss, since the deadlock occurs
before the database instance is actually closes.


-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

DO NOT REPLY [Bug 46218] deadlock on xindice+jetty shutdown

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46218





--- Comment #3 from Michael Gratton <mi...@vee.net>  2008-11-16 04:52:19 PST ---
I can't test it out right now since I had to put that machine back into
production for use tomorrow (Monday AEST).

To reproduce, on a reasonably fast dual-core machine, just continually start
and stop Xindice. I was effectively doing:

while true; do
   xindiced start
   sleep 5
   xindice ld -c /db/some/10k+collection
   xindice stop
   sleep 5
done

If this is still a problem, you'll see instances of xindice slowly piling up.
New ones can start even though the old ones are hanging around since the Jetty
handler will have stopped it listening on whatever port is is configured to do
so.

Note this might also indicate the "db.lock" database lock mechanism isn't
actually functioning.

What are the odds of this getting applied to 1.1 and having a 1.1.1 release? I
don't want to move to 1.2 on this machine yet.


-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.