You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Imesh Gunaratne <im...@apache.org> on 2014/04/17 21:42:27 UTC
Cause of Scale Down Logic Issue was Identified
Hi,
During last few days we noticed an intermittent issue in scale down logic
and it was reported in STRATOS-614.
Today after doing some thorough tests we identified the cause of this
issue. The problem was with the topology event receiver queue. Since this
queue was designed as a singleton, it was being shared among multiple event
receivers in Single JVM mode. As a result event messages were unevenly
being dequeued to message processors and some event receivers were missing
events randomly.
This problem was also there in Tenant event receiver, Health Stat event
receiver and Instance Notifier event receiver. I have now fixed this by
introducing a message queue for each event receiver instance created.
I have now committed this fix to master branch. Will do few more flow tests
to verify the correction and update with the results soon.
Thanks
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos
Re: Cause of Scale Down Logic Issue was Identified
Posted by Imesh Gunaratne <im...@apache.org>.
I have now verified the correction done in STRATOS-614 and the scale down
functionality is working as expected.
Thanks
On Fri, Apr 18, 2014 at 10:34 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Thanks Azeez! Yes, completely agree.
>
>
> On Fri, Apr 18, 2014 at 9:56 AM, Afkham Azeez <af...@gmail.com> wrote:
>
>>
>>
>>
>> On Fri, Apr 18, 2014 at 1:12 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>
>>> Hi,
>>>
>>> During last few days we noticed an intermittent issue in scale down
>>> logic and it was reported in STRATOS-614.
>>>
>>> Today after doing some thorough tests we identified the cause of this
>>> issue. The problem was with the topology event receiver queue. Since this
>>> queue was designed as a singleton,
>>>
>>
>> Singletons should be avoided as much as possible, and used with caution.
>> Some even consider it an anti-pattern.
>>
>>
>>> it was being shared among multiple event receivers in Single JVM mode.
>>> As a result event messages were unevenly being dequeued to message
>>> processors and some event receivers were missing events randomly.
>>>
>>> This problem was also there in Tenant event receiver, Health Stat event
>>> receiver and Instance Notifier event receiver. I have now fixed this by
>>> introducing a message queue for each event receiver instance created.
>>>
>>> I have now committed this fix to master branch. Will do few more flow
>>> tests to verify the correction and update with the results soon.
>>>
>>> Thanks
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PPMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> *Afkham Azeez*
>> Director of Architecture; WSO2, Inc.; http://wso2.com,
>> *Member; Apache Software Foundation; **http://www.apache.org/*<http://www.apache.org/>
>>
>> *email: **azeez@wso2.com* <az...@wso2.com>
>> * cell: +94 77 3320919 <%2B94%2077%203320919> blog: *
>> *http://blog.afkham.org* <http://blog.afkham.org>
>> *twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>> * linked-in: **http://lk.linkedin.com/in/afkhamazeez
>> <http://lk.linkedin.com/in/afkhamazeez>*
>>
>> *Lean . Enterprise . Middleware*
>>
>>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PPMC Member, Apache Stratos
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos
Re: Cause of Scale Down Logic Issue was Identified
Posted by Imesh Gunaratne <im...@apache.org>.
Thanks Azeez! Yes, completely agree.
On Fri, Apr 18, 2014 at 9:56 AM, Afkham Azeez <af...@gmail.com> wrote:
>
>
>
> On Fri, Apr 18, 2014 at 1:12 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Hi,
>>
>> During last few days we noticed an intermittent issue in scale down logic
>> and it was reported in STRATOS-614.
>>
>> Today after doing some thorough tests we identified the cause of this
>> issue. The problem was with the topology event receiver queue. Since this
>> queue was designed as a singleton,
>>
>
> Singletons should be avoided as much as possible, and used with caution.
> Some even consider it an anti-pattern.
>
>
>> it was being shared among multiple event receivers in Single JVM mode. As
>> a result event messages were unevenly being dequeued to message processors
>> and some event receivers were missing events randomly.
>>
>> This problem was also there in Tenant event receiver, Health Stat event
>> receiver and Instance Notifier event receiver. I have now fixed this by
>> introducing a message queue for each event receiver instance created.
>>
>> I have now committed this fix to master branch. Will do few more flow
>> tests to verify the correction and update with the results soon.
>>
>> Thanks
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PPMC Member, Apache Stratos
>>
>
>
>
> --
> *Afkham Azeez*
> Director of Architecture; WSO2, Inc.; http://wso2.com,
> *Member; Apache Software Foundation; **http://www.apache.org/*<http://www.apache.org/>
>
> *email: **azeez@wso2.com* <az...@wso2.com>
> * cell: +94 77 3320919 <%2B94%2077%203320919> blog: *
> *http://blog.afkham.org* <http://blog.afkham.org>
> *twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
> * linked-in: **http://lk.linkedin.com/in/afkhamazeez
> <http://lk.linkedin.com/in/afkhamazeez>*
>
> *Lean . Enterprise . Middleware*
>
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos
Re: Cause of Scale Down Logic Issue was Identified
Posted by Afkham Azeez <af...@gmail.com>.
On Fri, Apr 18, 2014 at 1:12 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi,
>
> During last few days we noticed an intermittent issue in scale down logic
> and it was reported in STRATOS-614.
>
> Today after doing some thorough tests we identified the cause of this
> issue. The problem was with the topology event receiver queue. Since this
> queue was designed as a singleton,
>
Singletons should be avoided as much as possible, and used with caution.
Some even consider it an anti-pattern.
> it was being shared among multiple event receivers in Single JVM mode. As
> a result event messages were unevenly being dequeued to message processors
> and some event receivers were missing events randomly.
>
> This problem was also there in Tenant event receiver, Health Stat event
> receiver and Instance Notifier event receiver. I have now fixed this by
> introducing a message queue for each event receiver instance created.
>
> I have now committed this fix to master branch. Will do few more flow
> tests to verify the correction and update with the results soon.
>
> Thanks
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PPMC Member, Apache Stratos
>
--
*Afkham Azeez*
Director of Architecture; WSO2, Inc.; http://wso2.com,
*Member; Apache Software Foundation;
**http://www.apache.org/*<http://www.apache.org/>
*email: **azeez@wso2.com* <az...@wso2.com>
* cell: +94 77 3320919 blog: **http://blog.afkham.org*<http://blog.afkham.org>
*twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
* linked-in: **http://lk.linkedin.com/in/afkhamazeez
<http://lk.linkedin.com/in/afkhamazeez>*
*Lean . Enterprise . Middleware*