You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aries.apache.org by "Victor Antonovich (JIRA)" <ji...@apache.org> on 2012/11/13 14:34:12 UTC

[jira] [Created] (ARIES-962) Deadlock while stopping OSGi container

Victor Antonovich created ARIES-962:
---------------------------------------

             Summary: Deadlock while stopping OSGi container
                 Key: ARIES-962
                 URL: https://issues.apache.org/jira/browse/ARIES-962
             Project: Aries
          Issue Type: Bug
          Components: Blueprint
    Affects Versions: blueprint-core-1.0.1
         Environment: Apache Karaf 2.3.0
            Reporter: Victor Antonovich


OSGi container (Apache Karaf) hangs while stopping due to the deadlock with next stacktrace:
{quote}
Found one Java-level deadlock:
=============================
"Blueprint Extender: 1":
  waiting to lock monitor 0x00007f92a0003d98 (object 0x00000000e0995860, a java.util.ArrayList),
  which is held by "FelixStartLevel"
"FelixStartLevel":
  waiting to lock monitor 0x00007f92a0003cf0 (object 0x00000000e09960f0, a java.lang.Object),
  which is held by "Blueprint Extender: 1"

Java stack information for the threads listed above:
===================================================
"Blueprint Extender: 1":
	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.getBestServiceReference(AbstractServiceReferenceRecipe.java:429)
	- waiting to lock <0x00000000e0995860> (a java.util.ArrayList)
	at org.apache.aries.blueprint.container.ReferenceRecipe.retrack(ReferenceRecipe.java:128)
	at org.apache.aries.blueprint.container.ReferenceRecipe.untrack(ReferenceRecipe.java:151)
	- locked <0x00000000e09960f0> (a java.lang.Object)
	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.serviceRemoved(AbstractServiceReferenceRecipe.java:346)
	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.access$200(AbstractServiceReferenceRecipe.java:71)
	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe$3.run(AbstractServiceReferenceRecipe.java:316)
	at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
"FelixStartLevel":
	at org.apache.aries.blueprint.container.ReferenceRecipe.doStop(ReferenceRecipe.java:122)
	- waiting to lock <0x00000000e09960f0> (a java.lang.Object)
	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.stop(AbstractServiceReferenceRecipe.java:153)
	- locked <0x00000000e0995860> (a java.util.ArrayList)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.untrackServiceReference(BlueprintContainerImpl.java:594)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.untrackServiceReferences(BlueprintContainerImpl.java:574)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:842)
	at org.apache.aries.blueprint.container.BlueprintExtender.destroyContext(BlueprintExtender.java:246)
	at org.apache.aries.blueprint.container.BlueprintExtender.bundleChanged(BlueprintExtender.java:238)
	at org.apache.aries.blueprint.container.BlueprintExtender$BlueprintBundleTrackerCustomizer.modifiedBundle(BlueprintExtender.java:434)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:198)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:128)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:468)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:161)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:117)
	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
	at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4244)
	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2351)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1214)
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:295)
	at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.
{quote}
It seems like Apache Aries problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ARIES-962) Deadlock while stopping OSGi container

Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/ARIES-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504631#comment-13504631 ] 

Guillaume Nodet commented on ARIES-962:
---------------------------------------

Experimenting with the following patch:
{code}

diff --git a/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/AbstractServiceReferenceRecipe.java b/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/AbstractServiceReferenceRecipe.java
index f9a0d3c..64cf93c 100644
--- a/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/AbstractServiceReferenceRecipe.java
+++ b/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/AbstractServiceReferenceRecipe.java
@@ -82,6 +82,7 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
     /** The list of listeners for this reference.  This list will be lazy created */
     protected List<Listener> listeners;
 
+    protected final Object monitor = new Object();
     private final List<ServiceReference> references = new ArrayList<ServiceReference>();
     private final AtomicBoolean started = new AtomicBoolean();
     private final AtomicBoolean satisfied = new AtomicBoolean();
@@ -128,7 +129,7 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
                 satisfied.set(optional);
                 // Synchronized block on references so that service events won't interfere with initial references tracking
                 // though this may not be sufficient because we don't control ordering of those events
-                synchronized (references) {
+                synchronized (monitor) {
                     getBundleContextForServiceLookup().addServiceListener(this, getOsgiFilter());
                     ServiceReference[] references = getBundleContextForServiceLookup().getServiceReferences(null, getOsgiFilter());
                     if (references != null) {
@@ -148,7 +149,7 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
 
     public void stop() {
         if (started.compareAndSet(true, false)) {
-            synchronized (references) {
+            synchronized (monitor) {
                 getBundleContextForServiceLookup().removeServiceListener(this);
                 doStop();
                 for (Iterator<ServiceReference> it = references.iterator(); it.hasNext();) {
@@ -324,7 +325,7 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
     private void serviceAdded(ServiceReference ref) {
         LOGGER.debug("Tracking reference {} for OSGi service {}", ref, getOsgiFilter());
         if (isStarted()) {
-            synchronized (references) {
+            synchronized (monitor) {
                 if (references.contains(ref)) {
                     return;
                 }
@@ -338,7 +339,7 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
     private void serviceModified(ServiceReference ref) {
         // ref must be in references and must be satisfied
         if (isStarted()) {
-            synchronized (references) {
+            synchronized (monitor) {
                 if (references.contains(ref)) {
                     track(ref);
                 }
@@ -351,7 +352,7 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
             LOGGER.debug("Untracking reference {} for OSGi service {}", ref, getOsgiFilter());
             boolean removed;
             boolean satisfied;
-            synchronized (references) {
+            synchronized (monitor) {
                 removed = references.remove(ref);
                 satisfied = optional || !references.isEmpty();
             }
@@ -433,13 +434,13 @@ public abstract class AbstractServiceReferenceRecipe extends AbstractRecipe impl
     }
     
     public List<ServiceReference> getServiceReferences() {
-        synchronized (references) {
+        synchronized (monitor) {
             return new ArrayList<ServiceReference>(references);
         }
     }
 
     public ServiceReference getBestServiceReference() {
-        synchronized (references) {
+        synchronized (monitor) {
             int length = references.size();
             if (length == 0) { /* if no service is being tracked */
                 return null;
diff --git a/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceListRecipe.java b/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceListRecipe.java
index f19d9ea..f1d3c29 100644
--- a/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceListRecipe.java
+++ b/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceListRecipe.java
@@ -50,8 +50,7 @@ public class ReferenceListRecipe extends AbstractServiceReferenceRecipe {
     private final List<ManagedCollection> collections = new ArrayList<ManagedCollection>();
     private final DynamicCollection<ServiceDispatcher> storage = new DynamicCollection<ServiceDispatcher>();
     private final List<ServiceDispatcher> unboundDispatchers = new ArrayList<ServiceDispatcher>();
-    private final Object monitor = new Object();
-    
+
     public ReferenceListRecipe(String name,
                          ExtendedBlueprintContainer blueprintContainer,
                          ReferenceListMetadata metadata,
diff --git a/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceRecipe.java b/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceRecipe.java
index 1c81586..0a1b2b2 100644
--- a/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceRecipe.java
+++ b/blueprint/blueprint-core/src/main/java/org/apache/aries/blueprint/container/ReferenceRecipe.java
@@ -61,7 +61,6 @@ public class ReferenceRecipe extends AbstractServiceReferenceRecipe {
     private volatile ServiceReference trackedServiceReference;
     private volatile Object trackedService;
     private Object defaultBean;
-    private final Object monitor = new Object();
 
     private final Collection<Class<?>> proxyChildBeanClasses;
     private final Collection<WeakReference<Voidable>> proxiedChildren;
{code}

                
> Deadlock while stopping OSGi container
> --------------------------------------
>
>                 Key: ARIES-962
>                 URL: https://issues.apache.org/jira/browse/ARIES-962
>             Project: Aries
>          Issue Type: Bug
>          Components: Blueprint
>    Affects Versions: blueprint-core-1.0.1
>         Environment: Apache Karaf 2.3.0
>            Reporter: Victor Antonovich
>            Assignee: Guillaume Nodet
>
> OSGi container (Apache Karaf) hangs while stopping due to the deadlock with next stacktrace:
> {quote}
> Found one Java-level deadlock:
> =============================
> "Blueprint Extender: 1":
>   waiting to lock monitor 0x00007f92a0003d98 (object 0x00000000e0995860, a java.util.ArrayList),
>   which is held by "FelixStartLevel"
> "FelixStartLevel":
>   waiting to lock monitor 0x00007f92a0003cf0 (object 0x00000000e09960f0, a java.lang.Object),
>   which is held by "Blueprint Extender: 1"
> Java stack information for the threads listed above:
> ===================================================
> "Blueprint Extender: 1":
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.getBestServiceReference(AbstractServiceReferenceRecipe.java:429)
> 	- waiting to lock <0x00000000e0995860> (a java.util.ArrayList)
> 	at org.apache.aries.blueprint.container.ReferenceRecipe.retrack(ReferenceRecipe.java:128)
> 	at org.apache.aries.blueprint.container.ReferenceRecipe.untrack(ReferenceRecipe.java:151)
> 	- locked <0x00000000e09960f0> (a java.lang.Object)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.serviceRemoved(AbstractServiceReferenceRecipe.java:346)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.access$200(AbstractServiceReferenceRecipe.java:71)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe$3.run(AbstractServiceReferenceRecipe.java:316)
> 	at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> "FelixStartLevel":
> 	at org.apache.aries.blueprint.container.ReferenceRecipe.doStop(ReferenceRecipe.java:122)
> 	- waiting to lock <0x00000000e09960f0> (a java.lang.Object)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.stop(AbstractServiceReferenceRecipe.java:153)
> 	- locked <0x00000000e0995860> (a java.util.ArrayList)
> 	at org.apache.aries.blueprint.container.BlueprintContainerImpl.untrackServiceReference(BlueprintContainerImpl.java:594)
> 	at org.apache.aries.blueprint.container.BlueprintContainerImpl.untrackServiceReferences(BlueprintContainerImpl.java:574)
> 	at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:842)
> 	at org.apache.aries.blueprint.container.BlueprintExtender.destroyContext(BlueprintExtender.java:246)
> 	at org.apache.aries.blueprint.container.BlueprintExtender.bundleChanged(BlueprintExtender.java:238)
> 	at org.apache.aries.blueprint.container.BlueprintExtender$BlueprintBundleTrackerCustomizer.modifiedBundle(BlueprintExtender.java:434)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:198)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:128)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:468)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:161)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:117)
> 	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
> 	at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
> 	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
> 	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4244)
> 	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2351)
> 	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1214)
> 	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:295)
> 	at java.lang.Thread.run(Thread.java:662)
> Found 1 deadlock.
> {quote}
> It seems like Apache Aries problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (ARIES-962) Deadlock while stopping OSGi container

Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/ARIES-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Guillaume Nodet reassigned ARIES-962:
-------------------------------------

    Assignee: Guillaume Nodet
    
> Deadlock while stopping OSGi container
> --------------------------------------
>
>                 Key: ARIES-962
>                 URL: https://issues.apache.org/jira/browse/ARIES-962
>             Project: Aries
>          Issue Type: Bug
>          Components: Blueprint
>    Affects Versions: blueprint-core-1.0.1
>         Environment: Apache Karaf 2.3.0
>            Reporter: Victor Antonovich
>            Assignee: Guillaume Nodet
>
> OSGi container (Apache Karaf) hangs while stopping due to the deadlock with next stacktrace:
> {quote}
> Found one Java-level deadlock:
> =============================
> "Blueprint Extender: 1":
>   waiting to lock monitor 0x00007f92a0003d98 (object 0x00000000e0995860, a java.util.ArrayList),
>   which is held by "FelixStartLevel"
> "FelixStartLevel":
>   waiting to lock monitor 0x00007f92a0003cf0 (object 0x00000000e09960f0, a java.lang.Object),
>   which is held by "Blueprint Extender: 1"
> Java stack information for the threads listed above:
> ===================================================
> "Blueprint Extender: 1":
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.getBestServiceReference(AbstractServiceReferenceRecipe.java:429)
> 	- waiting to lock <0x00000000e0995860> (a java.util.ArrayList)
> 	at org.apache.aries.blueprint.container.ReferenceRecipe.retrack(ReferenceRecipe.java:128)
> 	at org.apache.aries.blueprint.container.ReferenceRecipe.untrack(ReferenceRecipe.java:151)
> 	- locked <0x00000000e09960f0> (a java.lang.Object)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.serviceRemoved(AbstractServiceReferenceRecipe.java:346)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.access$200(AbstractServiceReferenceRecipe.java:71)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe$3.run(AbstractServiceReferenceRecipe.java:316)
> 	at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> "FelixStartLevel":
> 	at org.apache.aries.blueprint.container.ReferenceRecipe.doStop(ReferenceRecipe.java:122)
> 	- waiting to lock <0x00000000e09960f0> (a java.lang.Object)
> 	at org.apache.aries.blueprint.container.AbstractServiceReferenceRecipe.stop(AbstractServiceReferenceRecipe.java:153)
> 	- locked <0x00000000e0995860> (a java.util.ArrayList)
> 	at org.apache.aries.blueprint.container.BlueprintContainerImpl.untrackServiceReference(BlueprintContainerImpl.java:594)
> 	at org.apache.aries.blueprint.container.BlueprintContainerImpl.untrackServiceReferences(BlueprintContainerImpl.java:574)
> 	at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:842)
> 	at org.apache.aries.blueprint.container.BlueprintExtender.destroyContext(BlueprintExtender.java:246)
> 	at org.apache.aries.blueprint.container.BlueprintExtender.bundleChanged(BlueprintExtender.java:238)
> 	at org.apache.aries.blueprint.container.BlueprintExtender$BlueprintBundleTrackerCustomizer.modifiedBundle(BlueprintExtender.java:434)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:198)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:128)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:468)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:161)
> 	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:117)
> 	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
> 	at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
> 	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
> 	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4244)
> 	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2351)
> 	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1214)
> 	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:295)
> 	at java.lang.Thread.run(Thread.java:662)
> Found 1 deadlock.
> {quote}
> It seems like Apache Aries problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira