You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrija Panic <an...@gmail.com> on 2015/06/04 17:00:43 UTC

Strange bug? "spam" in management log files...

Hi,

I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes it
 happens that on the first node, we have extremem number of folowing line
entries in the log fie, which causes many GB log in just few hours or less:
(as you can see here they are not even that frequent, but sometimes, it
gets really crazy with the speed/numer logged per seconds:

2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
90520745449919: Resp: Routing to peer
2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
90520745449919: Resp: Routing to peer


We have haproxy VIP, to which SSVM connects, and all cloudstack agents
(agent.properties file).

Any suggestions, how to avoid this - I noticed when I turn off second ACS
MGMT server, and then reboot first one (restart cloudstack-management) it
stops and behaves nice :)

This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.

Thanks,
-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
I'm sorry for spaming,

it seems that in my db.properties file on second MGMT srever, I had
127.0.0.1 as the Cluster IP.
After this was changed to real IP address, it seems now that I dont have
those "spam" log messages, seems all fine for some hours.



On 5 June 2015 at 16:24, Andrija Panic <an...@gmail.com> wrote:

> Hi,
>
> any hint on how to proceed ?
>
> on haproxy I see rougly 50%/50% sessions across 2 backend servers.
>  But inside DB, it all points to the one mgmt_server_ip...
>
> Thanks,
> Andrija
>
> On 4 June 2015 at 19:27, Andrija Panic <an...@gmail.com> wrote:
>
>> And if of any help another hint:
>>
>> while Im having this lines sent to logs in high volume...if I stop second
>> mgmt server, first one (that is making all these lines, doesnt stop to make
>> them), so log is still heavily writen to - only when I also restart mgmt on
>> 1st node (2nd node is down), then these log lines dissapear.
>>
>> Thx
>>
>> On 4 June 2015 at 19:19, Andrija Panic <an...@gmail.com> wrote:
>>
>>> And I could add - these lines (in this volume) only appears on first
>>> mgmt server (Actually I have 2 separate, but identical ACS installations,
>>> and same behaviour).
>>>
>>> On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:
>>>
>>>> Just checked, in the HOSTS table, all agents are connected (via
>>>> haproxy) to the first mgmt server...I just restarted haproxy, and still
>>>> inside the DB, it says same mgmt_server_id for all agents - which is not
>>>> really true.
>>>>
>>>> Actually, on the haproxy itslef (statistics page) I can see almoust
>>>> 50%-50% distribution across 2 backends - which means by haproxy it should
>>>> be fine.
>>>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS
>>>> mgmt server)
>>>>
>>>> This is our haproxy config, I think it's fine, but... DB says
>>>> differently, althouh haproxy statistick say all fine
>>>>
>>>> ### ACS 8250
>>>> #######################################################################################
>>>> frontend front_ACS_8250 10.20.10.100:8250
>>>>         option tcplog
>>>>         mode tcp
>>>>         default_backend back_8250
>>>> backend back_8250
>>>>         mode tcp
>>>>         balance source
>>>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000
>>>> rise 3 fall 3
>>>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000
>>>> rise 3 fall 3
>>>>
>>>> ##################################################################################################
>>>>
>>>> Any info on how to proceed with this, since because of these lines, it
>>>> makes mgmt logs almoust unreadable... :(
>>>>
>>>> Thanks,
>>>> Andrija
>>>>
>>>> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>>>>
>>>>> Thanks Koushik,
>>>>>
>>>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>>>> expect this is expected :)
>>>>> I understand that the message is there because of setup, just an awful
>>>>> lot of lines....
>>>>>
>>>>> Will check thx for the help !
>>>>>
>>>>> Andrija
>>>>>
>>>>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>>>>
>>>>>> This is expected in a clustered MS setup. What is the distribution of
>>>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the
>>>>>> HV host processes all commands for that host.
>>>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in
>>>>>> both MS logs to correlate.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>>>> sometimes it
>>>>>> > happens that on the first node, we have extremem number of folowing
>>>>>> line
>>>>>> > entries in the log fie, which causes many GB log in just few hours
>>>>>> or less:
>>>>>> > (as you can see here they are not even that frequent, but
>>>>>> sometimes, it
>>>>>> > gets really crazy with the speed/numer logged per seconds:
>>>>>> >
>>>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> >
>>>>>> >
>>>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack
>>>>>> agents
>>>>>> > (agent.properties file).
>>>>>> >
>>>>>> > Any suggestions, how to avoid this - I noticed when I turn off
>>>>>> second ACS
>>>>>> > MGMT server, and then reboot first one (restart
>>>>>> cloudstack-management) it
>>>>>> > stops and behaves nice :)
>>>>>> >
>>>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>>>> >
>>>>>> > Thanks,
>>>>>> > --
>>>>>> >
>>>>>> > Andrija Panić
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Andrija Panić
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Andrija Panić
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
I'm sorry for spaming,

it seems that in my db.properties file on second MGMT srever, I had
127.0.0.1 as the Cluster IP.
After this was changed to real IP address, it seems now that I dont have
those "spam" log messages, seems all fine for some hours.



On 5 June 2015 at 16:24, Andrija Panic <an...@gmail.com> wrote:

> Hi,
>
> any hint on how to proceed ?
>
> on haproxy I see rougly 50%/50% sessions across 2 backend servers.
>  But inside DB, it all points to the one mgmt_server_ip...
>
> Thanks,
> Andrija
>
> On 4 June 2015 at 19:27, Andrija Panic <an...@gmail.com> wrote:
>
>> And if of any help another hint:
>>
>> while Im having this lines sent to logs in high volume...if I stop second
>> mgmt server, first one (that is making all these lines, doesnt stop to make
>> them), so log is still heavily writen to - only when I also restart mgmt on
>> 1st node (2nd node is down), then these log lines dissapear.
>>
>> Thx
>>
>> On 4 June 2015 at 19:19, Andrija Panic <an...@gmail.com> wrote:
>>
>>> And I could add - these lines (in this volume) only appears on first
>>> mgmt server (Actually I have 2 separate, but identical ACS installations,
>>> and same behaviour).
>>>
>>> On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:
>>>
>>>> Just checked, in the HOSTS table, all agents are connected (via
>>>> haproxy) to the first mgmt server...I just restarted haproxy, and still
>>>> inside the DB, it says same mgmt_server_id for all agents - which is not
>>>> really true.
>>>>
>>>> Actually, on the haproxy itslef (statistics page) I can see almoust
>>>> 50%-50% distribution across 2 backends - which means by haproxy it should
>>>> be fine.
>>>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS
>>>> mgmt server)
>>>>
>>>> This is our haproxy config, I think it's fine, but... DB says
>>>> differently, althouh haproxy statistick say all fine
>>>>
>>>> ### ACS 8250
>>>> #######################################################################################
>>>> frontend front_ACS_8250 10.20.10.100:8250
>>>>         option tcplog
>>>>         mode tcp
>>>>         default_backend back_8250
>>>> backend back_8250
>>>>         mode tcp
>>>>         balance source
>>>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000
>>>> rise 3 fall 3
>>>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000
>>>> rise 3 fall 3
>>>>
>>>> ##################################################################################################
>>>>
>>>> Any info on how to proceed with this, since because of these lines, it
>>>> makes mgmt logs almoust unreadable... :(
>>>>
>>>> Thanks,
>>>> Andrija
>>>>
>>>> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>>>>
>>>>> Thanks Koushik,
>>>>>
>>>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>>>> expect this is expected :)
>>>>> I understand that the message is there because of setup, just an awful
>>>>> lot of lines....
>>>>>
>>>>> Will check thx for the help !
>>>>>
>>>>> Andrija
>>>>>
>>>>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>>>>
>>>>>> This is expected in a clustered MS setup. What is the distribution of
>>>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the
>>>>>> HV host processes all commands for that host.
>>>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in
>>>>>> both MS logs to correlate.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>>>> sometimes it
>>>>>> > happens that on the first node, we have extremem number of folowing
>>>>>> line
>>>>>> > entries in the log fie, which causes many GB log in just few hours
>>>>>> or less:
>>>>>> > (as you can see here they are not even that frequent, but
>>>>>> sometimes, it
>>>>>> > gets really crazy with the speed/numer logged per seconds:
>>>>>> >
>>>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> >
>>>>>> >
>>>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack
>>>>>> agents
>>>>>> > (agent.properties file).
>>>>>> >
>>>>>> > Any suggestions, how to avoid this - I noticed when I turn off
>>>>>> second ACS
>>>>>> > MGMT server, and then reboot first one (restart
>>>>>> cloudstack-management) it
>>>>>> > stops and behaves nice :)
>>>>>> >
>>>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>>>> >
>>>>>> > Thanks,
>>>>>> > --
>>>>>> >
>>>>>> > Andrija Panić
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Andrija Panić
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Andrija Panić
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
Hi,

any hint on how to proceed ?

on haproxy I see rougly 50%/50% sessions across 2 backend servers.
 But inside DB, it all points to the one mgmt_server_ip...

Thanks,
Andrija

On 4 June 2015 at 19:27, Andrija Panic <an...@gmail.com> wrote:

> And if of any help another hint:
>
> while Im having this lines sent to logs in high volume...if I stop second
> mgmt server, first one (that is making all these lines, doesnt stop to make
> them), so log is still heavily writen to - only when I also restart mgmt on
> 1st node (2nd node is down), then these log lines dissapear.
>
> Thx
>
> On 4 June 2015 at 19:19, Andrija Panic <an...@gmail.com> wrote:
>
>> And I could add - these lines (in this volume) only appears on first mgmt
>> server (Actually I have 2 separate, but identical ACS installations, and
>> same behaviour).
>>
>> On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:
>>
>>> Just checked, in the HOSTS table, all agents are connected (via haproxy)
>>> to the first mgmt server...I just restarted haproxy, and still inside the
>>> DB, it says same mgmt_server_id for all agents - which is not really true.
>>>
>>> Actually, on the haproxy itslef (statistics page) I can see almoust
>>> 50%-50% distribution across 2 backends - which means by haproxy it should
>>> be fine.
>>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
>>> server)
>>>
>>> This is our haproxy config, I think it's fine, but... DB says
>>> differently, althouh haproxy statistick say all fine
>>>
>>> ### ACS 8250
>>> #######################################################################################
>>> frontend front_ACS_8250 10.20.10.100:8250
>>>         option tcplog
>>>         mode tcp
>>>         default_backend back_8250
>>> backend back_8250
>>>         mode tcp
>>>         balance source
>>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000
>>> rise 3 fall 3
>>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000
>>> rise 3 fall 3
>>>
>>> ##################################################################################################
>>>
>>> Any info on how to proceed with this, since because of these lines, it
>>> makes mgmt logs almoust unreadable... :(
>>>
>>> Thanks,
>>> Andrija
>>>
>>> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>>>
>>>> Thanks Koushik,
>>>>
>>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>>> expect this is expected :)
>>>> I understand that the message is there because of setup, just an awful
>>>> lot of lines....
>>>>
>>>> Will check thx for the help !
>>>>
>>>> Andrija
>>>>
>>>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>>>
>>>>> This is expected in a clustered MS setup. What is the distribution of
>>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the
>>>>> HV host processes all commands for that host.
>>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in
>>>>> both MS logs to correlate.
>>>>>
>>>>>
>>>>>
>>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>>> sometimes it
>>>>> > happens that on the first node, we have extremem number of folowing
>>>>> line
>>>>> > entries in the log fie, which causes many GB log in just few hours
>>>>> or less:
>>>>> > (as you can see here they are not even that frequent, but sometimes,
>>>>> it
>>>>> > gets really crazy with the speed/numer logged per seconds:
>>>>> >
>>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> >
>>>>> >
>>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack
>>>>> agents
>>>>> > (agent.properties file).
>>>>> >
>>>>> > Any suggestions, how to avoid this - I noticed when I turn off
>>>>> second ACS
>>>>> > MGMT server, and then reboot first one (restart
>>>>> cloudstack-management) it
>>>>> > stops and behaves nice :)
>>>>> >
>>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>>> >
>>>>> > Thanks,
>>>>> > --
>>>>> >
>>>>> > Andrija Panić
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Andrija Panić
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
Hi,

any hint on how to proceed ?

on haproxy I see rougly 50%/50% sessions across 2 backend servers.
 But inside DB, it all points to the one mgmt_server_ip...

Thanks,
Andrija

On 4 June 2015 at 19:27, Andrija Panic <an...@gmail.com> wrote:

> And if of any help another hint:
>
> while Im having this lines sent to logs in high volume...if I stop second
> mgmt server, first one (that is making all these lines, doesnt stop to make
> them), so log is still heavily writen to - only when I also restart mgmt on
> 1st node (2nd node is down), then these log lines dissapear.
>
> Thx
>
> On 4 June 2015 at 19:19, Andrija Panic <an...@gmail.com> wrote:
>
>> And I could add - these lines (in this volume) only appears on first mgmt
>> server (Actually I have 2 separate, but identical ACS installations, and
>> same behaviour).
>>
>> On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:
>>
>>> Just checked, in the HOSTS table, all agents are connected (via haproxy)
>>> to the first mgmt server...I just restarted haproxy, and still inside the
>>> DB, it says same mgmt_server_id for all agents - which is not really true.
>>>
>>> Actually, on the haproxy itslef (statistics page) I can see almoust
>>> 50%-50% distribution across 2 backends - which means by haproxy it should
>>> be fine.
>>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
>>> server)
>>>
>>> This is our haproxy config, I think it's fine, but... DB says
>>> differently, althouh haproxy statistick say all fine
>>>
>>> ### ACS 8250
>>> #######################################################################################
>>> frontend front_ACS_8250 10.20.10.100:8250
>>>         option tcplog
>>>         mode tcp
>>>         default_backend back_8250
>>> backend back_8250
>>>         mode tcp
>>>         balance source
>>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000
>>> rise 3 fall 3
>>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000
>>> rise 3 fall 3
>>>
>>> ##################################################################################################
>>>
>>> Any info on how to proceed with this, since because of these lines, it
>>> makes mgmt logs almoust unreadable... :(
>>>
>>> Thanks,
>>> Andrija
>>>
>>> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>>>
>>>> Thanks Koushik,
>>>>
>>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>>> expect this is expected :)
>>>> I understand that the message is there because of setup, just an awful
>>>> lot of lines....
>>>>
>>>> Will check thx for the help !
>>>>
>>>> Andrija
>>>>
>>>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>>>
>>>>> This is expected in a clustered MS setup. What is the distribution of
>>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the
>>>>> HV host processes all commands for that host.
>>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in
>>>>> both MS logs to correlate.
>>>>>
>>>>>
>>>>>
>>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>>> sometimes it
>>>>> > happens that on the first node, we have extremem number of folowing
>>>>> line
>>>>> > entries in the log fie, which causes many GB log in just few hours
>>>>> or less:
>>>>> > (as you can see here they are not even that frequent, but sometimes,
>>>>> it
>>>>> > gets really crazy with the speed/numer logged per seconds:
>>>>> >
>>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>>> > 90520745449919: Resp: Routing to peer
>>>>> >
>>>>> >
>>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack
>>>>> agents
>>>>> > (agent.properties file).
>>>>> >
>>>>> > Any suggestions, how to avoid this - I noticed when I turn off
>>>>> second ACS
>>>>> > MGMT server, and then reboot first one (restart
>>>>> cloudstack-management) it
>>>>> > stops and behaves nice :)
>>>>> >
>>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>>> >
>>>>> > Thanks,
>>>>> > --
>>>>> >
>>>>> > Andrija Panić
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Andrija Panić
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
And if of any help another hint:

while Im having this lines sent to logs in high volume...if I stop second
mgmt server, first one (that is making all these lines, doesnt stop to make
them), so log is still heavily writen to - only when I also restart mgmt on
1st node (2nd node is down), then these log lines dissapear.

Thx

On 4 June 2015 at 19:19, Andrija Panic <an...@gmail.com> wrote:

> And I could add - these lines (in this volume) only appears on first mgmt
> server (Actually I have 2 separate, but identical ACS installations, and
> same behaviour).
>
> On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:
>
>> Just checked, in the HOSTS table, all agents are connected (via haproxy)
>> to the first mgmt server...I just restarted haproxy, and still inside the
>> DB, it says same mgmt_server_id for all agents - which is not really true.
>>
>> Actually, on the haproxy itslef (statistics page) I can see almoust
>> 50%-50% distribution across 2 backends - which means by haproxy it should
>> be fine.
>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
>> server)
>>
>> This is our haproxy config, I think it's fine, but... DB says
>> differently, althouh haproxy statistick say all fine
>>
>> ### ACS 8250
>> #######################################################################################
>> frontend front_ACS_8250 10.20.10.100:8250
>>         option tcplog
>>         mode tcp
>>         default_backend back_8250
>> backend back_8250
>>         mode tcp
>>         balance source
>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise
>> 3 fall 3
>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise
>> 3 fall 3
>>
>> ##################################################################################################
>>
>> Any info on how to proceed with this, since because of these lines, it
>> makes mgmt logs almoust unreadable... :(
>>
>> Thanks,
>> Andrija
>>
>> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>>
>>> Thanks Koushik,
>>>
>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>> expect this is expected :)
>>> I understand that the message is there because of setup, just an awful
>>> lot of lines....
>>>
>>> Will check thx for the help !
>>>
>>> Andrija
>>>
>>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>>
>>>> This is expected in a clustered MS setup. What is the distribution of
>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the
>>>> HV host processes all commands for that host.
>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>>>> MS logs to correlate.
>>>>
>>>>
>>>>
>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>> sometimes it
>>>> > happens that on the first node, we have extremem number of folowing
>>>> line
>>>> > entries in the log fie, which causes many GB log in just few hours or
>>>> less:
>>>> > (as you can see here they are not even that frequent, but sometimes,
>>>> it
>>>> > gets really crazy with the speed/numer logged per seconds:
>>>> >
>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> >
>>>> >
>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>>>> > (agent.properties file).
>>>> >
>>>> > Any suggestions, how to avoid this - I noticed when I turn off second
>>>> ACS
>>>> > MGMT server, and then reboot first one (restart
>>>> cloudstack-management) it
>>>> > stops and behaves nice :)
>>>> >
>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>> >
>>>> > Thanks,
>>>> > --
>>>> >
>>>> > Andrija Panić
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
And if of any help another hint:

while Im having this lines sent to logs in high volume...if I stop second
mgmt server, first one (that is making all these lines, doesnt stop to make
them), so log is still heavily writen to - only when I also restart mgmt on
1st node (2nd node is down), then these log lines dissapear.

Thx

On 4 June 2015 at 19:19, Andrija Panic <an...@gmail.com> wrote:

> And I could add - these lines (in this volume) only appears on first mgmt
> server (Actually I have 2 separate, but identical ACS installations, and
> same behaviour).
>
> On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:
>
>> Just checked, in the HOSTS table, all agents are connected (via haproxy)
>> to the first mgmt server...I just restarted haproxy, and still inside the
>> DB, it says same mgmt_server_id for all agents - which is not really true.
>>
>> Actually, on the haproxy itslef (statistics page) I can see almoust
>> 50%-50% distribution across 2 backends - which means by haproxy it should
>> be fine.
>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
>> server)
>>
>> This is our haproxy config, I think it's fine, but... DB says
>> differently, althouh haproxy statistick say all fine
>>
>> ### ACS 8250
>> #######################################################################################
>> frontend front_ACS_8250 10.20.10.100:8250
>>         option tcplog
>>         mode tcp
>>         default_backend back_8250
>> backend back_8250
>>         mode tcp
>>         balance source
>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise
>> 3 fall 3
>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise
>> 3 fall 3
>>
>> ##################################################################################################
>>
>> Any info on how to proceed with this, since because of these lines, it
>> makes mgmt logs almoust unreadable... :(
>>
>> Thanks,
>> Andrija
>>
>> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>>
>>> Thanks Koushik,
>>>
>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>> expect this is expected :)
>>> I understand that the message is there because of setup, just an awful
>>> lot of lines....
>>>
>>> Will check thx for the help !
>>>
>>> Andrija
>>>
>>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>>
>>>> This is expected in a clustered MS setup. What is the distribution of
>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the
>>>> HV host processes all commands for that host.
>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>>>> MS logs to correlate.
>>>>
>>>>
>>>>
>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>> sometimes it
>>>> > happens that on the first node, we have extremem number of folowing
>>>> line
>>>> > entries in the log fie, which causes many GB log in just few hours or
>>>> less:
>>>> > (as you can see here they are not even that frequent, but sometimes,
>>>> it
>>>> > gets really crazy with the speed/numer logged per seconds:
>>>> >
>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>> > 90520745449919: Resp: Routing to peer
>>>> >
>>>> >
>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>>>> > (agent.properties file).
>>>> >
>>>> > Any suggestions, how to avoid this - I noticed when I turn off second
>>>> ACS
>>>> > MGMT server, and then reboot first one (restart
>>>> cloudstack-management) it
>>>> > stops and behaves nice :)
>>>> >
>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>> >
>>>> > Thanks,
>>>> > --
>>>> >
>>>> > Andrija Panić
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
And I could add - these lines (in this volume) only appears on first mgmt
server (Actually I have 2 separate, but identical ACS installations, and
same behaviour).

On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:

> Just checked, in the HOSTS table, all agents are connected (via haproxy)
> to the first mgmt server...I just restarted haproxy, and still inside the
> DB, it says same mgmt_server_id for all agents - which is not really true.
>
> Actually, on the haproxy itslef (statistics page) I can see almoust
> 50%-50% distribution across 2 backends - which means by haproxy it should
> be fine.
> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
> server)
>
> This is our haproxy config, I think it's fine, but... DB says differently,
> althouh haproxy statistick say all fine
>
> ### ACS 8250
> #######################################################################################
> frontend front_ACS_8250 10.20.10.100:8250
>         option tcplog
>         mode tcp
>         default_backend back_8250
> backend back_8250
>         mode tcp
>         balance source
>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise
> 3 fall 3
>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise
> 3 fall 3
>
> ##################################################################################################
>
> Any info on how to proceed with this, since because of these lines, it
> makes mgmt logs almoust unreadable... :(
>
> Thanks,
> Andrija
>
> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>
>> Thanks Koushik,
>>
>> I will check and let you know - but 11GB log file for 10h ?  I dont
>> expect this is expected :)
>> I understand that the message is there because of setup, just an awful
>> lot of lines....
>>
>> Will check thx for the help !
>>
>> Andrija
>>
>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>
>>> This is expected in a clustered MS setup. What is the distribution of HV
>>> hosts across these MS (check host table in db for MS id)? MS owning the HV
>>> host processes all commands for that host.
>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>>> MS logs to correlate.
>>>
>>>
>>>
>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>> sometimes it
>>> > happens that on the first node, we have extremem number of folowing
>>> line
>>> > entries in the log fie, which causes many GB log in just few hours or
>>> less:
>>> > (as you can see here they are not even that frequent, but sometimes, it
>>> > gets really crazy with the speed/numer logged per seconds:
>>> >
>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> >
>>> >
>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>>> > (agent.properties file).
>>> >
>>> > Any suggestions, how to avoid this - I noticed when I turn off second
>>> ACS
>>> > MGMT server, and then reboot first one (restart cloudstack-management)
>>> it
>>> > stops and behaves nice :)
>>> >
>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>> >
>>> > Thanks,
>>> > --
>>> >
>>> > Andrija Panić
>>>
>>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
And I could add - these lines (in this volume) only appears on first mgmt
server (Actually I have 2 separate, but identical ACS installations, and
same behaviour).

On 4 June 2015 at 19:18, Andrija Panic <an...@gmail.com> wrote:

> Just checked, in the HOSTS table, all agents are connected (via haproxy)
> to the first mgmt server...I just restarted haproxy, and still inside the
> DB, it says same mgmt_server_id for all agents - which is not really true.
>
> Actually, on the haproxy itslef (statistics page) I can see almoust
> 50%-50% distribution across 2 backends - which means by haproxy it should
> be fine.
> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
> server)
>
> This is our haproxy config, I think it's fine, but... DB says differently,
> althouh haproxy statistick say all fine
>
> ### ACS 8250
> #######################################################################################
> frontend front_ACS_8250 10.20.10.100:8250
>         option tcplog
>         mode tcp
>         default_backend back_8250
> backend back_8250
>         mode tcp
>         balance source
>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise
> 3 fall 3
>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise
> 3 fall 3
>
> ##################################################################################################
>
> Any info on how to proceed with this, since because of these lines, it
> makes mgmt logs almoust unreadable... :(
>
> Thanks,
> Andrija
>
> On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:
>
>> Thanks Koushik,
>>
>> I will check and let you know - but 11GB log file for 10h ?  I dont
>> expect this is expected :)
>> I understand that the message is there because of setup, just an awful
>> lot of lines....
>>
>> Will check thx for the help !
>>
>> Andrija
>>
>> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>>
>>> This is expected in a clustered MS setup. What is the distribution of HV
>>> hosts across these MS (check host table in db for MS id)? MS owning the HV
>>> host processes all commands for that host.
>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>>> MS logs to correlate.
>>>
>>>
>>>
>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>> sometimes it
>>> > happens that on the first node, we have extremem number of folowing
>>> line
>>> > entries in the log fie, which causes many GB log in just few hours or
>>> less:
>>> > (as you can see here they are not even that frequent, but sometimes, it
>>> > gets really crazy with the speed/numer logged per seconds:
>>> >
>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>> > 90520745449919: Resp: Routing to peer
>>> >
>>> >
>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>>> > (agent.properties file).
>>> >
>>> > Any suggestions, how to avoid this - I noticed when I turn off second
>>> ACS
>>> > MGMT server, and then reboot first one (restart cloudstack-management)
>>> it
>>> > stops and behaves nice :)
>>> >
>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>> >
>>> > Thanks,
>>> > --
>>> >
>>> > Andrija Panić
>>>
>>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
Just checked, in the HOSTS table, all agents are connected (via haproxy) to
the first mgmt server...I just restarted haproxy, and still inside the DB,
it says same mgmt_server_id for all agents - which is not really true.

Actually, on the haproxy itslef (statistics page) I can see almoust 50%-50%
distribution across 2 backends - which means by haproxy it should be fine.
total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
server)

This is our haproxy config, I think it's fine, but... DB says differently,
althouh haproxy statistick say all fine

### ACS 8250
#######################################################################################
frontend front_ACS_8250 10.20.10.100:8250
        option tcplog
        mode tcp
        default_backend back_8250
backend back_8250
        mode tcp
        balance source
        server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise 3
fall 3
        server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise 3
fall 3
##################################################################################################

Any info on how to proceed with this, since because of these lines, it
makes mgmt logs almoust unreadable... :(

Thanks,
Andrija

On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:

> Thanks Koushik,
>
> I will check and let you know - but 11GB log file for 10h ?  I dont expect
> this is expected :)
> I understand that the message is there because of setup, just an awful lot
> of lines....
>
> Will check thx for the help !
>
> Andrija
>
> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>
>> This is expected in a clustered MS setup. What is the distribution of HV
>> hosts across these MS (check host table in db for MS id)? MS owning the HV
>> host processes all commands for that host.
>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>> MS logs to correlate.
>>
>>
>>
>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes
>> it
>> > happens that on the first node, we have extremem number of folowing line
>> > entries in the log fie, which causes many GB log in just few hours or
>> less:
>> > (as you can see here they are not even that frequent, but sometimes, it
>> > gets really crazy with the speed/numer logged per seconds:
>> >
>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> >
>> >
>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>> > (agent.properties file).
>> >
>> > Any suggestions, how to avoid this - I noticed when I turn off second
>> ACS
>> > MGMT server, and then reboot first one (restart cloudstack-management)
>> it
>> > stops and behaves nice :)
>> >
>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>> >
>> > Thanks,
>> > --
>> >
>> > Andrija Panić
>>
>>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
Just checked, in the HOSTS table, all agents are connected (via haproxy) to
the first mgmt server...I just restarted haproxy, and still inside the DB,
it says same mgmt_server_id for all agents - which is not really true.

Actually, on the haproxy itslef (statistics page) I can see almoust 50%-50%
distribution across 2 backends - which means by haproxy it should be fine.
total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt
server)

This is our haproxy config, I think it's fine, but... DB says differently,
althouh haproxy statistick say all fine

### ACS 8250
#######################################################################################
frontend front_ACS_8250 10.20.10.100:8250
        option tcplog
        mode tcp
        default_backend back_8250
backend back_8250
        mode tcp
        balance source
        server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 rise 3
fall 3
        server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 rise 3
fall 3
##################################################################################################

Any info on how to proceed with this, since because of these lines, it
makes mgmt logs almoust unreadable... :(

Thanks,
Andrija

On 4 June 2015 at 19:00, Andrija Panic <an...@gmail.com> wrote:

> Thanks Koushik,
>
> I will check and let you know - but 11GB log file for 10h ?  I dont expect
> this is expected :)
> I understand that the message is there because of setup, just an awful lot
> of lines....
>
> Will check thx for the help !
>
> Andrija
>
> On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:
>
>> This is expected in a clustered MS setup. What is the distribution of HV
>> hosts across these MS (check host table in db for MS id)? MS owning the HV
>> host processes all commands for that host.
>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both
>> MS logs to correlate.
>>
>>
>>
>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes
>> it
>> > happens that on the first node, we have extremem number of folowing line
>> > entries in the log fie, which causes many GB log in just few hours or
>> less:
>> > (as you can see here they are not even that frequent, but sometimes, it
>> > gets really crazy with the speed/numer logged per seconds:
>> >
>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>> > 90520745449919: Resp: Routing to peer
>> >
>> >
>> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
>> > (agent.properties file).
>> >
>> > Any suggestions, how to avoid this - I noticed when I turn off second
>> ACS
>> > MGMT server, and then reboot first one (restart cloudstack-management)
>> it
>> > stops and behaves nice :)
>> >
>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>> >
>> > Thanks,
>> > --
>> >
>> > Andrija Panić
>>
>>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
Thanks Koushik,

I will check and let you know - but 11GB log file for 10h ?  I dont expect
this is expected :)
I understand that the message is there because of setup, just an awful lot
of lines....

Will check thx for the help !

Andrija

On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:

> This is expected in a clustered MS setup. What is the distribution of HV
> hosts across these MS (check host table in db for MS id)? MS owning the HV
> host processes all commands for that host.
> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS
> logs to correlate.
>
>
>
> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com> wrote:
>
> > Hi,
> >
> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes
> it
> > happens that on the first node, we have extremem number of folowing line
> > entries in the log fie, which causes many GB log in just few hours or
> less:
> > (as you can see here they are not even that frequent, but sometimes, it
> > gets really crazy with the speed/numer logged per seconds:
> >
> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> >
> >
> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
> > (agent.properties file).
> >
> > Any suggestions, how to avoid this - I noticed when I turn off second ACS
> > MGMT server, and then reboot first one (restart cloudstack-management) it
> > stops and behaves nice :)
> >
> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
> >
> > Thanks,
> > --
> >
> > Andrija Panić
>
>


-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Andrija Panic <an...@gmail.com>.
Thanks Koushik,

I will check and let you know - but 11GB log file for 10h ?  I dont expect
this is expected :)
I understand that the message is there because of setup, just an awful lot
of lines....

Will check thx for the help !

Andrija

On 4 June 2015 at 18:53, Koushik Das <ko...@citrix.com> wrote:

> This is expected in a clustered MS setup. What is the distribution of HV
> hosts across these MS (check host table in db for MS id)? MS owning the HV
> host processes all commands for that host.
> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS
> logs to correlate.
>
>
>
> On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com> wrote:
>
> > Hi,
> >
> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes
> it
> > happens that on the first node, we have extremem number of folowing line
> > entries in the log fie, which causes many GB log in just few hours or
> less:
> > (as you can see here they are not even that frequent, but sometimes, it
> > gets really crazy with the speed/numer logged per seconds:
> >
> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
> > 90520745449919: Resp: Routing to peer
> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
> > 90520745449919: Resp: Routing to peer
> >
> >
> > We have haproxy VIP, to which SSVM connects, and all cloudstack agents
> > (agent.properties file).
> >
> > Any suggestions, how to avoid this - I noticed when I turn off second ACS
> > MGMT server, and then reboot first one (restart cloudstack-management) it
> > stops and behaves nice :)
> >
> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
> >
> > Thanks,
> > --
> >
> > Andrija Panić
>
>


-- 

Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Koushik Das <ko...@citrix.com>.
This is expected in a clustered MS setup. What is the distribution of HV hosts across these MS (check host table in db for MS id)? MS owning the HV host processes all commands for that host.
Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS logs to correlate.



On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com> wrote:

> Hi,
> 
> I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes it
> happens that on the first node, we have extremem number of folowing line
> entries in the log fie, which causes many GB log in just few hours or less:
> (as you can see here they are not even that frequent, but sometimes, it
> gets really crazy with the speed/numer logged per seconds:
> 
> 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 
> 
> We have haproxy VIP, to which SSVM connects, and all cloudstack agents
> (agent.properties file).
> 
> Any suggestions, how to avoid this - I noticed when I turn off second ACS
> MGMT server, and then reboot first one (restart cloudstack-management) it
> stops and behaves nice :)
> 
> This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
> 
> Thanks,
> -- 
> 
> Andrija Panić

Re: Strange bug? "spam" in management log files...

Posted by Koushik Das <ko...@citrix.com>.
This is expected in a clustered MS setup. What is the distribution of HV hosts across these MS (check host table in db for MS id)? MS owning the HV host processes all commands for that host.
Grep for the sequence numbers (for e.g. 73-7374644389819187201) in both MS logs to correlate.



On 04-Jun-2015, at 8:30 PM, Andrija Panic <an...@gmail.com> wrote:

> Hi,
> 
> I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and sometimes it
> happens that on the first node, we have extremem number of folowing line
> entries in the log fie, which causes many GB log in just few hours or less:
> (as you can see here they are not even that frequent, but sometimes, it
> gets really crazy with the speed/numer logged per seconds:
> 
> 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
> 90520745449919: Resp: Routing to peer
> 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
> 90520745449919: Resp: Routing to peer
> 
> 
> We have haproxy VIP, to which SSVM connects, and all cloudstack agents
> (agent.properties file).
> 
> Any suggestions, how to avoid this - I noticed when I turn off second ACS
> MGMT server, and then reboot first one (restart cloudstack-management) it
> stops and behaves nice :)
> 
> This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
> 
> Thanks,
> -- 
> 
> Andrija Panić