Balancer Persist testing

[Rainer Jung <ra...@kippdata.de>]

I played around with the current backport proposal for balancer
persistence, ie. persisting config changes applied to proxy balancer via
the Balancer Manager GUI interface to disk and reloading them during
startup.

I made some observations. My setup uses a globally defined balancer with
two members and another two globally defined ones with each one member.
All members are different. Platform is Solaris 10 Sparc.

I also use two vhosts, but those are only the targets of the members.

Further local setting:

ProxyStatus Full
BalancerPersist On
ProxyPassInherit Off (also in the vhosts)

Findings:

- The "Used" count in balancer manager drops to "0" after restart/reboot
with persist on. It seems the slot management is not completely right.
Not yet further investigated.

- A warning might be needed about mixing changes via balancer manager
and via config:

  - If changing the path for a balancer in config while persist is on
    and then restarting, the balancer manager shows the old path
    from the persisted shm, but it seems the new one from the
    config is in effect.

  - E.g. setting a member state to stopped in config and again doing
    a persist restart, the old state from shm wins and the change in
    config is ignored.

So it seems before applying changes to the config one should always
backport all changes applied via the manager (not trivial, because you'd
have to check all displayed values against the configured ones), then
stop, then remove the *.persist files, apply config changes and start again.

- Switching persistence from on to off in config and then doing a
restart keeps the old .persist files around. They are ignored during the
startup as expected, but next time you'll switch back to persist on, the
outdated .persist file will again be loaded.

- "Elected" display in balancer manager survives restart, whereas
  "To" and "From" (byte counts) do not.

Findings for proxy unrelated to persistance:

- sometimes the busy display in balancer manager shows a "1" instead of
"0" after an ab test run. Often it goes down again to "0" after the next
test run.

- when using a high concurrency in ab the global server (reverse proxy)
which forwards to two vhosts reports

AH01102: error reading status line from remote server apache:9092

and I couldn't yet track it down to either keep alive, process recycles
or dynamic listener deactivation.

Regards,

Rainer

Re: Balancer Persist testing

[Mladen Turk <mt...@apache.org>]

On 12/14/2012 05:15 PM, Rainer Jung wrote:
> On 14.12.2012 17:11, Mladen Turk wrote:
>> On 12/14/2012 01:10 PM, Rainer Jung wrote:
>>>
>>> - The "Used" count in balancer manager drops to "0" after restart/reboot
>>> with persist on. It seems the slot management is not completely right.
>>> Not yet further investigated.
>>>
>>
>> IMHO this should actually be a desired behaviour.
>> I suppose one would expect to have a 'clean room' after restart.
>
> Please note that "Used" isn't what it sounds like. It is not a use counter.
>
> It indicates how many slots in slotmem are taken by balancer member
> instances, e.g. if your balancer has two members, the "Used" shows "2"
> before restart and "0" after.
>

Ah, I thought you were referring to runtime statistical data.


Regards
-- 
^TM

Re: Balancer Persist testing

[Rainer Jung <ra...@kippdata.de>]

On 14.12.2012 17:11, Mladen Turk wrote:
> On 12/14/2012 01:10 PM, Rainer Jung wrote:
>>
>> - The "Used" count in balancer manager drops to "0" after restart/reboot
>> with persist on. It seems the slot management is not completely right.
>> Not yet further investigated.
>>
> 
> IMHO this should actually be a desired behaviour.
> I suppose one would expect to have a 'clean room' after restart.

Please note that "Used" isn't what it sounds like. It is not a use counter.

It indicates how many slots in slotmem are taken by balancer member
instances, e.g. if your balancer has two members, the "Used" shows "2"
before restart and "0" after.

Regards,

Rainer

Re: Balancer Persist testing

[Mladen Turk <mt...@apache.org>]

On 12/14/2012 01:10 PM, Rainer Jung wrote:
>
> - The "Used" count in balancer manager drops to "0" after restart/reboot
> with persist on. It seems the slot management is not completely right.
> Not yet further investigated.
>

IMHO this should actually be a desired behaviour.
I suppose one would expect to have a 'clean room' after restart.



Regards
-- 
^TM

Re: Balancer Persist testing

[Jim Jagielski <ji...@jaguNET.com>]

Yeah... basically, whenever we change the configuration,
we need to "reset" all the counters which are used for
any of the lbmethods, since:

  1. The old stats are no longer applicable, and would
     invalidate the lbmethod
  2. In the process of adding/subtracting workers or
     balancers, we may have also changed lbmethods.

Other data fields are more "statistical" values, and so
retaining them may, or may not, have value... either retaining
or clearing them is fine with me; I have no strong prefs one
way or the other.

On Dec 14, 2012, at 8:36 AM, Rainer Jung <ra...@kippdata.de> wrote:

> On 14.12.2012 13:39, Jim Jagielski wrote:
>> 
>> On Dec 14, 2012, at 7:10 AM, Rainer Jung <ra...@kippdata.de> wrote:
>>> 
>>> - The "Used" count in balancer manager drops to "0" after restart/reboot
>>> with persist on. It seems the slot management is not completely right.
>>> Not yet further investigated.
>> 
>> This is intentional actually. The configs are maintained, but
>> the stats are not, since we must start the "counting" from
>> scratch since the config has changed.
> 
> Note that "Used" isn't the request count. It is the number of slots
> already occupied with data (so typically it shows the number of members
> of the balancer). If dropping it to "0" after restart is intended,
> what's the meaning of the field then?
> 
>>> - A warning might be needed about mixing changes via balancer manager
>>> and via config:
>>> 
>>> - If changing the path for a balancer in config while persist is on
>>>   and then restarting, the balancer manager shows the old path
>>>   from the persisted shm, but it seems the new one from the
>>>   config is in effect.
>>> 
>>> - E.g. setting a member state to stopped in config and again doing
>>>   a persist restart, the old state from shm wins and the change in
>>>   config is ignored.
>>> 
>> +1
>> 
>>> So it seems before applying changes to the config one should always
>>> backport all changes applied via the manager (not trivial, because you'd
>>> have to check all displayed values against the configured ones), then
>>> stop, then remove the *.persist files, apply config changes and start again.
>>> 
>>> - Switching persistence from on to off in config and then doing a
>>> restart keeps the old .persist files around. They are ignored during the
>>> startup as expected, but next time you'll switch back to persist on, the
>>> outdated .persist file will again be loaded.
>>> 
>> 
>> Yeah, good point...
>> 
>>> - "Elected" display in balancer manager survives restart, whereas
>>> "To" and "From" (byte counts) do not.
>>> 
>> 
>> Again intentional.
> 
>> From the above I got the impression you wanted to not persist stats. So
> why not also drop "Elected"? Is it, because "Elected" by itself is never
> used for a balancing decision, whereas "From" and "To" are in case
> "bytraffic" is used? Then I would understand the reason technically, but
> for users it will be a bit confusing to persist the number of requests
> but not the traffic. I would then prefer to not persist the number of
> requests as well. Otherwise the data shown after restart is inconsistent.
> 
> Very nice feature, but hard to get it right!
> 
> Regards,
> 
> Rainer
> 

Re: Balancer Persist testing

[Rainer Jung <ra...@kippdata.de>]

On 14.12.2012 13:39, Jim Jagielski wrote:
> 
> On Dec 14, 2012, at 7:10 AM, Rainer Jung <ra...@kippdata.de> wrote:
>>
>> - The "Used" count in balancer manager drops to "0" after restart/reboot
>> with persist on. It seems the slot management is not completely right.
>> Not yet further investigated.
> 
> This is intentional actually. The configs are maintained, but
> the stats are not, since we must start the "counting" from
> scratch since the config has changed.

Note that "Used" isn't the request count. It is the number of slots
already occupied with data (so typically it shows the number of members
of the balancer). If dropping it to "0" after restart is intended,
what's the meaning of the field then?

>> - A warning might be needed about mixing changes via balancer manager
>> and via config:
>>
>>  - If changing the path for a balancer in config while persist is on
>>    and then restarting, the balancer manager shows the old path
>>    from the persisted shm, but it seems the new one from the
>>    config is in effect.
>>
>>  - E.g. setting a member state to stopped in config and again doing
>>    a persist restart, the old state from shm wins and the change in
>>    config is ignored.
>>
> +1
> 
>> So it seems before applying changes to the config one should always
>> backport all changes applied via the manager (not trivial, because you'd
>> have to check all displayed values against the configured ones), then
>> stop, then remove the *.persist files, apply config changes and start again.
>>
>> - Switching persistence from on to off in config and then doing a
>> restart keeps the old .persist files around. They are ignored during the
>> startup as expected, but next time you'll switch back to persist on, the
>> outdated .persist file will again be loaded.
>>
> 
> Yeah, good point...
> 
>> - "Elected" display in balancer manager survives restart, whereas
>>  "To" and "From" (byte counts) do not.
>>
> 
> Again intentional.

>From the above I got the impression you wanted to not persist stats. So
why not also drop "Elected"? Is it, because "Elected" by itself is never
used for a balancing decision, whereas "From" and "To" are in case
"bytraffic" is used? Then I would understand the reason technically, but
for users it will be a bit confusing to persist the number of requests
but not the traffic. I would then prefer to not persist the number of
requests as well. Otherwise the data shown after restart is inconsistent.

Very nice feature, but hard to get it right!

Regards,

Rainer

Re: Balancer Persist testing

[Jim Jagielski <ji...@jaguNET.com>]

On Dec 14, 2012, at 7:10 AM, Rainer Jung <ra...@kippdata.de> wrote:
> 
> - The "Used" count in balancer manager drops to "0" after restart/reboot
> with persist on. It seems the slot management is not completely right.
> Not yet further investigated.

This is intentional actually. The configs are maintained, but
the stats are not, since we must start the "counting" from
scratch since the config has changed.
> 
> - A warning might be needed about mixing changes via balancer manager
> and via config:
> 
>  - If changing the path for a balancer in config while persist is on
>    and then restarting, the balancer manager shows the old path
>    from the persisted shm, but it seems the new one from the
>    config is in effect.
> 
>  - E.g. setting a member state to stopped in config and again doing
>    a persist restart, the old state from shm wins and the change in
>    config is ignored.
> 
+1

> So it seems before applying changes to the config one should always
> backport all changes applied via the manager (not trivial, because you'd
> have to check all displayed values against the configured ones), then
> stop, then remove the *.persist files, apply config changes and start again.
> 
> - Switching persistence from on to off in config and then doing a
> restart keeps the old .persist files around. They are ignored during the
> startup as expected, but next time you'll switch back to persist on, the
> outdated .persist file will again be loaded.
> 

Yeah, good point...

> - "Elected" display in balancer manager survives restart, whereas
>  "To" and "From" (byte counts) do not.
> 

Again intentional.