PATCH for deadlocking issues with LoadedObjectsRegistry use of WeakHashMap

[Galvin Hsiu <gk...@yahoo.com>]

Hi Thomas,

Both Joe Elliott and I did encounter this issue and
investigated it at length by examining the source code
to the WeakHashMap.  What we discovered was a resource
contention issue in the WeakHashMap implementation on
JDK 1.4.1_03 (not sure if this was fixed in the 
previous JDK versions or versions after) - but
whenever puts are getting called concurrently on the
same WeakHashMap, the while loop in the put function
call of the Map interface gets stuck since the said
iterator is modified by another thread.  Let me know
if you want further explanation and I will try to post
up the said code snippet.

The patch is as follows - synchronize using the weak
hash map on puts, remove and clears, but not on the
gets. These critical sections will ensure only 1
thread can modify this weak hash map at a time; I
personally think the performance impact on this is
minimal since the LoadedObjectRegistry seems to be
utilized PRIMARILY (aka explicit lookups) for the
frameworks utilizing the OTM layer (JDO, ODMG) as
opposed to the PB layer, which we are using right now.
Regardless, I'd rather trade a little performance hit
as opposed to ballooning CPU usage / thrashing due to
this interesting bug.

// BEGIN PATCHED FILE

package org.apache.ojb.broker.core;

/*
====================================================================
 * The Apache Software License, Version 1.1
 *
 * Copyright (c) 2001 The Apache Software Foundation. 
All rights
 * reserved.
 *
 * Redistribution and use in source and binary forms,
with or without
 * modification, are permitted provided that the
following conditions
 * are met:
 *
 * 1. Redistributions of source code must retain the
above copyright
 *    notice, this list of conditions and the
following disclaimer.
 *
 * 2. Redistributions in binary form must reproduce
the above copyright
 *    notice, this list of conditions and the
following disclaimer in
 *    the documentation and/or other materials
provided with the
 *    distribution.
 *
 * 3. The end-user documentation included with the
redistribution,
 *    if any, must include the following
acknowledgment:
 *       "This product includes software developed by
the
 *        Apache Software Foundation
(http://www.apache.org/)."
 *    Alternately, this acknowledgment may appear in
the software itself,
 *    if and wherever such third-party acknowledgments
normally appear.
 *
 * 4. The names "Apache" and "Apache Software
Foundation" and
 *    "Apache ObjectRelationalBridge" must not be used
to endorse or promote products
 *    derived from this software without prior written
permission. For
 *    written permission, please contact
apache@apache.org.
 *
 * 5. Products derived from this software may not be
called "Apache",
 *    "Apache ObjectRelationalBridge", nor may
"Apache" appear in their name, without
 *    prior written permission of the Apache Software
Foundation.
 *
 * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY
EXPRESSED OR IMPLIED
 * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES
 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE
 * DISCLAIMED.  IN NO EVENT SHALL THE APACHE SOFTWARE
FOUNDATION OR
 * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL,
 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT
 * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF
 * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND
 * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY,
 * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT
 * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF
 * SUCH DAMAGE.
 *
====================================================================
 *
 * This software consists of voluntary contributions
made by many
 * individuals on behalf of the Apache Software
Foundation.  For more
 * information on the Apache Software Foundation,
please see
 * <http://www.apache.org/>.
 */

import java.util.WeakHashMap;

/**
 * This is a helper class which registers all objects
loaded
 * from database. It is used by ODMG layer to
determine the state of
 * objects: if an object was not loaded from database
then it is new.
 * Note: objects remain registered even after they are
deleted. This
 * is necessary to prevent creation of deleted objects
by another
 * thread, see <a
href="http://archives.apache.org/eyebrowse/ReadMsg?listId=106&msgNo=1382">this<a>
 * for details.
 *
 * @author <a href="mailto:olegnitz@apache.org">Oleg
Nitz<a>
 * @version $Id: LoadedObjectsRegistry.java,v 1.1
2003/04/26 23:18:25 arminw Exp $
 */
public class LoadedObjectsRegistry
{
    private static WeakHashMap registry = new
WeakHashMap();

    private static final Object DUMMY = new Object();

    public static void register(Object object)
    {
        // Patch: critical section on the put of the
registry
        synchronized(registry)
        {
        registry.put(object, DUMMY);
        }
    }

    public static boolean isRegistered(Object object)
    {
        return (registry.get(object) != null);
    }

    public static void clear()
    {
        // Patch: critical section on the clear of the
registry
        synchronized(registry)
        {
        registry.clear();
        }
    }

    public static void remove(Object object)
    {
        if(object != null)
        {
        // Patch: critical section on the remove of
the registry
            synchronized(registry)
            {
            registry.remove(object);
            }
        }
    }
}



Hi Joseph,

To include your changes into the codebase please post
your patch (or 
patched files).
YOu can post it as a zipped file either to the list or
directly to me.
Of course we want to get OJB free of such issues!

cheers,
Thomas

Joseph Elliott wrote:
> has anybody else encountered deadlocking issues
under load with the ojb
> library?  we started seeing frequent situations with
pegged cpu usage and
> stuck threads testing a large web application
(running under tomcat on duel
> cpu linux boxes)  a thread dump showed numerous
threads all stuck in the
> WeakHashMap.put's called from
LoadedObjectsRegistry.java
> 
> adding synchronize blocks around the
LoadedObjectsRegistry.java put and
> clear calls has seemed to have stabilized things. 
does any body have any
> opinions about the performance impact of this and if
there are any other
> alternatives to making the system safe under highly
concurrent usage?
> 
> we're currently working with a patched version of
ojb for our application,
> but it would be nice if this could be addressed in
the official code base
> for the next rc.
> 
> thanks much,
>  - joe





__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com