[jira] [Created] (BROOKLYN-121) Rebind fails (too many dangling references) after terminating apps that used port-forwaring

["Aled Sage (JIRA)" <ji...@apache.org>]

Aled Sage created BROOKLYN-121:
----------------------------------

             Summary: Rebind fails (too many dangling references) after terminating apps that used port-forwaring
                 Key: BROOKLYN-121
                 URL: https://issues.apache.org/jira/browse/BROOKLYN-121
             Project: Brooklyn
          Issue Type: Bug
    Affects Versions: 0.7.0-SNAPSHOT
            Reporter: Aled Sage


Use of subnet tiers with port-forwarding can cause problems on rebind after the apps have been terminated: you can get an error like "Failure rebinding: Too many dangling references: 45 of 607"

The `brooklyn.location.access.PortForwardManager` tracks the port-forwarding that has been set up. This includes holding a reference to the location object (in `brooklyn.location.access.PortMapping`). If the entity is unmanaged without explicitly removing the port mapping, then the `PortForwardManger` keeps a reference to the location object even though it is no longer managed! On rebind, it cannot resolve the reference so it is counted as a "dangling reference".

Some code was added a while ago to work around issues with cloud providers' object stores, where we only got back a subset of the objects in a container (without any warnings about it). This previously caused Brooklyn to rebind as best it could, discarding the dangling references. However, doing that was dangerous: it meant some apps disappeared without trace; better to fail and tell the operator so they can investigate. In this cloud's case, trying again a few minutes later resolved the problem.

The distraction above was to explain why we fail if there are "too many dangling references".

A workaround is to disable the dangling reference count check by adding something like the following to your brooklyn.properties:

{noformat}
rebind.failureMode.danglingRefs.minRequiredHealthy=[[0,0],[2,0]]
{noformat}

The format for this configuration value is explained at the javadoc of `brooklyn.util.collections.QuorumCheck.QuorumChecks.newLinearRange()`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)