You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Rajkumar Rajaratnam <ra...@wso2.com> on 2015/11/13 05:53:25 UTC
[Stratos 4.1.5] [Blocker] Monitors are not created on Stratos restart
Hi Devs,
Experiencing $Subject.
Root cause is, AS received the complete-topology-event before loading
application-contexts from registry to in-memory model. So the
complete-topology-event is ignored by AS, but the topology is initialized.
This also means that all the upcoming periodic complete-topology-events
will be ignored by AS. Refer the following code block in
AutoscalerTopologyEventReceiver.
private void addEventListeners() {
// Listen to topology events that affect clusters
topologyEventReceiver.addEventListener(new CompleteTopologyEventListener() {
@Override
protected void onEvent(Event event) {
if (!topologyInitialized) {
log.info("[CompleteTopologyEvent] Received: " +
event.getClass());
try {
ApplicationHolder.acquireReadLock();
Applications applications =
ApplicationHolder.getApplications();
if (applications != null) {
for (Application application : applications.
getApplications().values()) {
ApplicationContext applicationContext =
AutoscalerContext.getInstance().
getApplicationContext(application.getUniqueIdentifier());
if (applicationContext != null &&
applicationContext.getStatus().
equals(ApplicationContext.STATUS_DEPLOYED)) {
if
(AutoscalerUtil.allClustersInitialized(application)) {
AutoscalerUtil.getInstance().startApplicationMonitor(
application.getUniqueIdentifier());
} else {
log.error("Complete Topology is
not consistent with " +
"the applications which
got persisted");
}
} else {
log.info("The application is not yet " +
"deployed for this [application] " +
application.getUniqueIdentifier());
}
}
topologyInitialized = true;
} else {
log.info("No applications found in the
complete topology");
}
} catch (Exception e) {
log.error("Error processing event", e);
} finally {
ApplicationHolder.releaseReadLock();
}
}
}
});
This code block should ensure that the monitors should be created for all
the persisted applications. Currently, it is not ensuring this - I am
fixing it.
I guess, there are two ways to fix it.
a) Topology should be not initialized in the above code block
(topologyInitialized = true) until application contexts are loaded from
registry.
b) Complete topology request event should be not be sent until application
contexts are loaded from registry.
Thanks,
Raj.
--
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2
Mobile : +94777568639
Blog : rajkumarr.com
Re: [Stratos 4.1.5] [Blocker] Monitors are not created on Stratos restart
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
Fixed in 5c63e93026c580b373dbb513c1ae1c
On Fri, Nov 13, 2015 at 10:53 AM, Isuru Haththotuwa <is...@apache.org>
wrote:
>
>
> On Fri, Nov 13, 2015 at 10:46 AM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
>
>> Nice catch Raj!
>>
>> I think better to wait until contexts/policies are loaded from registry
>> before starting AutoscalerTopologyReceiver. We're seeing this issue now
>> because topology is initialized immediately after starting the receiver
>> with topology initializing optimization.
>>
> +1. Lets start the TopologyRecvr thread after we load the information from
> the registry. Otherwise there is no way to be sure what will happen since
> the two processes are happening simultaneously.
>
>>
>> Thanks.
>>
>> On Fri, Nov 13, 2015 at 10:23 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>> > wrote:
>>
>>> Hi Devs,
>>>
>>> Experiencing $Subject.
>>>
>>> Root cause is, AS received the complete-topology-event before loading
>>> application-contexts from registry to in-memory model. So the
>>> complete-topology-event is ignored by AS, but the topology is initialized.
>>> This also means that all the upcoming periodic complete-topology-events
>>> will be ignored by AS. Refer the following code block in
>>> AutoscalerTopologyEventReceiver.
>>>
>>> private void addEventListeners() {
>>> // Listen to topology events that affect clusters
>>> topologyEventReceiver.addEventListener(new CompleteTopologyEventListener() {
>>> @Override
>>> protected void onEvent(Event event) {
>>> if (!topologyInitialized) {
>>> log.info("[CompleteTopologyEvent] Received: " + event.getClass());
>>> try {
>>> ApplicationHolder.acquireReadLock();
>>> Applications applications = ApplicationHolder.getApplications();
>>> if (applications != null) {
>>> for (Application application : applications.
>>> getApplications().values()) {
>>> ApplicationContext applicationContext =
>>> AutoscalerContext.getInstance().
>>> getApplicationContext(application.getUniqueIdentifier());
>>> if (applicationContext != null && applicationContext.getStatus().
>>> equals(ApplicationContext.STATUS_DEPLOYED)) {
>>> if (AutoscalerUtil.allClustersInitialized(application)) {
>>> AutoscalerUtil.getInstance().startApplicationMonitor(
>>> application.getUniqueIdentifier());
>>> } else {
>>> log.error("Complete Topology is not consistent with " +
>>> "the applications which got persisted");
>>> }
>>> } else {
>>> log.info("The application is not yet " +
>>> "deployed for this [application] " +
>>> application.getUniqueIdentifier());
>>> }
>>>
>>> }
>>> topologyInitialized = true;
>>> } else {
>>> log.info("No applications found in the complete topology");
>>> }
>>> } catch (Exception e) {
>>> log.error("Error processing event", e);
>>> } finally {
>>> ApplicationHolder.releaseReadLock();
>>> }
>>> }
>>> }
>>> });
>>>
>>>
>>> This code block should ensure that the monitors should be created for
>>> all the persisted applications. Currently, it is not ensuring this - I am
>>> fixing it.
>>>
>>> I guess, there are two ways to fix it.
>>>
>>> a) Topology should be not initialized in the above code block
>>> (topologyInitialized = true) until application contexts are loaded from
>>> registry.
>>> b) Complete topology request event should be not be sent until
>>> application contexts are loaded from registry.
>>>
>>> Thanks,
>>> Raj.
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Committer & PMC Member, Apache Stratos
>>> Software Engineer, WSO2
>>>
>>> Mobile : +94777568639
>>> Blog : rajkumarr.com
>>>
>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.; http://wso2.com/
>>
>> Blog:
>> http://ravihansa3000.blogspot.com
>>
>> --
>> <http://ravihansa3000.blogspot.com>
>> <http://ravihansa3000.blogspot.com>
>> Thanks and Regards,
>>
>> Isuru H.
>> <http://ravihansa3000.blogspot.com>
>> +94 716 358 048 <http://ravihansa3000.blogspot.com>* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>
--
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2
Mobile : +94777568639
Blog : rajkumarr.com
Re: [Stratos 4.1.5] [Blocker] Monitors are not created on Stratos restart
Posted by Isuru Haththotuwa <is...@apache.org>.
On Fri, Nov 13, 2015 at 10:46 AM, Akila Ravihansa Perera <ravihansa@wso2.com
> wrote:
> Nice catch Raj!
>
> I think better to wait until contexts/policies are loaded from registry
> before starting AutoscalerTopologyReceiver. We're seeing this issue now
> because topology is initialized immediately after starting the receiver
> with topology initializing optimization.
>
+1. Lets start the TopologyRecvr thread after we load the information from
the registry. Otherwise there is no way to be sure what will happen since
the two processes are happening simultaneously.
>
> Thanks.
>
> On Fri, Nov 13, 2015 at 10:23 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi Devs,
>>
>> Experiencing $Subject.
>>
>> Root cause is, AS received the complete-topology-event before loading
>> application-contexts from registry to in-memory model. So the
>> complete-topology-event is ignored by AS, but the topology is initialized.
>> This also means that all the upcoming periodic complete-topology-events
>> will be ignored by AS. Refer the following code block in
>> AutoscalerTopologyEventReceiver.
>>
>> private void addEventListeners() {
>> // Listen to topology events that affect clusters
>> topologyEventReceiver.addEventListener(new CompleteTopologyEventListener() {
>> @Override
>> protected void onEvent(Event event) {
>> if (!topologyInitialized) {
>> log.info("[CompleteTopologyEvent] Received: " + event.getClass());
>> try {
>> ApplicationHolder.acquireReadLock();
>> Applications applications = ApplicationHolder.getApplications();
>> if (applications != null) {
>> for (Application application : applications.
>> getApplications().values()) {
>> ApplicationContext applicationContext =
>> AutoscalerContext.getInstance().
>> getApplicationContext(application.getUniqueIdentifier());
>> if (applicationContext != null && applicationContext.getStatus().
>> equals(ApplicationContext.STATUS_DEPLOYED)) {
>> if (AutoscalerUtil.allClustersInitialized(application)) {
>> AutoscalerUtil.getInstance().startApplicationMonitor(
>> application.getUniqueIdentifier());
>> } else {
>> log.error("Complete Topology is not consistent with " +
>> "the applications which got persisted");
>> }
>> } else {
>> log.info("The application is not yet " +
>> "deployed for this [application] " +
>> application.getUniqueIdentifier());
>> }
>>
>> }
>> topologyInitialized = true;
>> } else {
>> log.info("No applications found in the complete topology");
>> }
>> } catch (Exception e) {
>> log.error("Error processing event", e);
>> } finally {
>> ApplicationHolder.releaseReadLock();
>> }
>> }
>> }
>> });
>>
>>
>> This code block should ensure that the monitors should be created for all
>> the persisted applications. Currently, it is not ensuring this - I am
>> fixing it.
>>
>> I guess, there are two ways to fix it.
>>
>> a) Topology should be not initialized in the above code block
>> (topologyInitialized = true) until application contexts are loaded from
>> registry.
>> b) Complete topology request event should be not be sent until
>> application contexts are loaded from registry.
>>
>> Thanks,
>> Raj.
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.; http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>
> --
> <http://ravihansa3000.blogspot.com>
> <http://ravihansa3000.blogspot.com>
> Thanks and Regards,
>
> Isuru H.
> <http://ravihansa3000.blogspot.com>
> +94 716 358 048 <http://ravihansa3000.blogspot.com>* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>
Re: [Stratos 4.1.5] [Blocker] Monitors are not created on Stratos restart
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Nice catch Raj!
I think better to wait until contexts/policies are loaded from registry
before starting AutoscalerTopologyReceiver. We're seeing this issue now
because topology is initialized immediately after starting the receiver
with topology initializing optimization.
Thanks.
On Fri, Nov 13, 2015 at 10:23 AM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi Devs,
>
> Experiencing $Subject.
>
> Root cause is, AS received the complete-topology-event before loading
> application-contexts from registry to in-memory model. So the
> complete-topology-event is ignored by AS, but the topology is initialized.
> This also means that all the upcoming periodic complete-topology-events
> will be ignored by AS. Refer the following code block in
> AutoscalerTopologyEventReceiver.
>
> private void addEventListeners() {
> // Listen to topology events that affect clusters
> topologyEventReceiver.addEventListener(new CompleteTopologyEventListener() {
> @Override
> protected void onEvent(Event event) {
> if (!topologyInitialized) {
> log.info("[CompleteTopologyEvent] Received: " + event.getClass());
> try {
> ApplicationHolder.acquireReadLock();
> Applications applications = ApplicationHolder.getApplications();
> if (applications != null) {
> for (Application application : applications.
> getApplications().values()) {
> ApplicationContext applicationContext =
> AutoscalerContext.getInstance().
> getApplicationContext(application.getUniqueIdentifier());
> if (applicationContext != null && applicationContext.getStatus().
> equals(ApplicationContext.STATUS_DEPLOYED)) {
> if (AutoscalerUtil.allClustersInitialized(application)) {
> AutoscalerUtil.getInstance().startApplicationMonitor(
> application.getUniqueIdentifier());
> } else {
> log.error("Complete Topology is not consistent with " +
> "the applications which got persisted");
> }
> } else {
> log.info("The application is not yet " +
> "deployed for this [application] " +
> application.getUniqueIdentifier());
> }
>
> }
> topologyInitialized = true;
> } else {
> log.info("No applications found in the complete topology");
> }
> } catch (Exception e) {
> log.error("Error processing event", e);
> } finally {
> ApplicationHolder.releaseReadLock();
> }
> }
> }
> });
>
>
> This code block should ensure that the monitors should be created for all
> the persisted applications. Currently, it is not ensuring this - I am
> fixing it.
>
> I guess, there are two ways to fix it.
>
> a) Topology should be not initialized in the above code block
> (topologyInitialized = true) until application contexts are loaded from
> registry.
> b) Complete topology request event should be not be sent until application
> contexts are loaded from registry.
>
> Thanks,
> Raj.
>
> --
> Rajkumar Rajaratnam
> Committer & PMC Member, Apache Stratos
> Software Engineer, WSO2
>
> Mobile : +94777568639
> Blog : rajkumarr.com
>
--
Akila Ravihansa Perera
WSO2 Inc.; http://wso2.com/
Blog: http://ravihansa3000.blogspot.com