You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com> on 2014/11/14 01:49:09 UTC

Un-subscribe cartridge takes 60+ sec

Hi,

While testing Stratos 4.1 M3, I noticed about half the times, unsubscribing a cartridge takes 60+ seconds to execute.  During that time, the caller is blocked.  I poke around Stratos code and found TopicPublisher class that is responsible for the 60 second retry delay.  The code sleeps for 60 seconds whenever we encounter an exception while posting the given message to its corresponding topic.   Looks like this code has been around since at least release 4.0.0.

It seems to me 60 sec delay is way to long.  If you have a lot of subscriptions, the delay can be multiple factor of 60 seconds.   Also, if you do this via Stratos' Rest API, the request might get timed out before the response comes back?

I reduced this delay from 60 seconds to one second and tested and noticed that worked just fine.   I'm planning to push this change upstream unless I get any objection on that.

As for why the exception while attempting to update the topic, I did some googling and found [1].   It seems this is an issue with mqtt-client.   Somehow we're getting a stale connection to MB when we update the topic.   We're currently using version 0.4.0 of mqtt-client.  From [2], it looks like the latest version is 1.0.0.   Maybe it's time to upgrade?

Other question: when I make a Rest call to get list of subscriptions and it returns nothing, is it safe to assume all spawn instances have been killed?

Regards,
-Jeffrey

[1] https://github.com/openhab/openhab/issues/980
[2] http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/

Re: Un-subscribe cartridge takes 60+ sec

Posted by Imesh Gunaratne <im...@apache.org>.
Hi Jeffrey,

Seems like mqtt-client 1.0.0 maven artifact is not still published:
https://repo.eclipse.org/content/repositories/paho-releases/org/eclipse/paho/mqtt-client/

Thanks

On Sat, Nov 15, 2014 at 12:36 AM, Jeffrey Nguyen (jeffrngu) <
jeffrngu@cisco.com> wrote:

>
>  Thanks Udara and Isuru for your replies.   I like the approach of doing
> retry with incremental delay
>
>  Hi Imesh,
>
>  Please go ahead and implement this fix and also take care of the
> mqtt-client upgrade.
>
>  Thanks,
> -Jeffrey
>
>   From: Imesh Gunaratne <im...@apache.org>
> Reply-To: "dev@stratos.apache.org" <de...@stratos.apache.org>
> Date: Friday, November 14, 2014 11:18 AM
> To: dev <de...@stratos.apache.org>
> Subject: Re: Un-subscribe cartridge takes 60+ sec
>
>    Hi Jeffrey,
>
> Thanks for bringing this up. I was thinking about this issue sometime back
> and thought that may be we could start the retry interval with a small
> value and then increase it step by step depending on the number of
> continuous failures. I noticed this approach in Gmail.
>
>  I can do a quick fix for this if you don't mind.
>
>  Of course, we could upgrade the mqtt-client version. Thanks for pointing
> this out! I will do this.
>
>  Thanks
>
> On Fri, Nov 14, 2014 at 6:19 AM, Jeffrey Nguyen (jeffrngu) <
> jeffrngu@cisco.com> wrote:
>
>>  Hi,
>>
>>  While testing Stratos 4.1 M3, I noticed about half the times,
>> unsubscribing a cartridge takes 60+ seconds to execute.  During that time,
>> the caller is blocked.  I poke around Stratos code and found TopicPublisher
>> class that is responsible for the 60 second retry delay.  The code sleeps
>> for 60 seconds whenever we encounter an exception while posting the given
>> message to its corresponding topic.   Looks like this code has been around
>> since at least release 4.0.0.
>>
>>  It seems to me 60 sec delay is way to long.  If you have a lot of
>> subscriptions, the delay can be multiple factor of 60 seconds.   Also, if
>> you do this via Stratos' Rest API, the request might get timed out before
>> the response comes back?
>>
>>  I reduced this delay from 60 seconds to one second and tested and
>> noticed that worked just fine.   I'm planning to push this change upstream
>> unless I get any objection on that.
>>
>>  As for why the exception while attempting to update the topic, I did
>> some googling and found [1].   It seems this is an issue with mqtt-client.
>>   Somehow we're getting a stale connection to MB when we update the topic.
>>   We're currently using version 0.4.0 of mqtt-client.  From [2], it looks
>> like the latest version is 1.0.0.   Maybe it's time to upgrade?
>>
>>  Other question: when I make a Rest call to get list of subscriptions
>> and it returns nothing, is it safe to assume all spawn instances have been
>> killed?
>>
>>  Regards,
>> -Jeffrey
>>
>>  [1] https://github.com/openhab/openhab/issues/980
>> *[2] *http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/
>>
>
>
>
> --
>  Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: Un-subscribe cartridge takes 60+ sec

Posted by "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>.
Thanks Udara and Isuru for your replies.   I like the approach of doing retry with incremental delay

Hi Imesh,

Please go ahead and implement this fix and also take care of the mqtt-client upgrade.

Thanks,
-Jeffrey

From: Imesh Gunaratne <im...@apache.org>>
Reply-To: "dev@stratos.apache.org<ma...@stratos.apache.org>" <de...@stratos.apache.org>>
Date: Friday, November 14, 2014 11:18 AM
To: dev <de...@stratos.apache.org>>
Subject: Re: Un-subscribe cartridge takes 60+ sec

Hi Jeffrey,

Thanks for bringing this up. I was thinking about this issue sometime back and thought that may be we could start the retry interval with a small value and then increase it step by step depending on the number of continuous failures. I noticed this approach in Gmail.

I can do a quick fix for this if you don't mind.

Of course, we could upgrade the mqtt-client version. Thanks for pointing this out! I will do this.

Thanks

On Fri, Nov 14, 2014 at 6:19 AM, Jeffrey Nguyen (jeffrngu) <je...@cisco.com>> wrote:
Hi,

While testing Stratos 4.1 M3, I noticed about half the times, unsubscribing a cartridge takes 60+ seconds to execute.  During that time, the caller is blocked.  I poke around Stratos code and found TopicPublisher class that is responsible for the 60 second retry delay.  The code sleeps for 60 seconds whenever we encounter an exception while posting the given message to its corresponding topic.   Looks like this code has been around since at least release 4.0.0.

It seems to me 60 sec delay is way to long.  If you have a lot of subscriptions, the delay can be multiple factor of 60 seconds.   Also, if you do this via Stratos' Rest API, the request might get timed out before the response comes back?

I reduced this delay from 60 seconds to one second and tested and noticed that worked just fine.   I'm planning to push this change upstream unless I get any objection on that.

As for why the exception while attempting to update the topic, I did some googling and found [1].   It seems this is an issue with mqtt-client.   Somehow we're getting a stale connection to MB when we update the topic.   We're currently using version 0.4.0 of mqtt-client.  From [2], it looks like the latest version is 1.0.0.   Maybe it's time to upgrade?

Other question: when I make a Rest call to get list of subscriptions and it returns nothing, is it safe to assume all spawn instances have been killed?

Regards,
-Jeffrey

[1] https://github.com/openhab/openhab/issues/980
[2] http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: Un-subscribe cartridge takes 60+ sec

Posted by Imesh Gunaratne <im...@apache.org>.
Hi Jeffrey,

Thanks for bringing this up. I was thinking about this issue sometime back
and thought that may be we could start the retry interval with a small
value and then increase it step by step depending on the number of
continuous failures. I noticed this approach in Gmail.

I can do a quick fix for this if you don't mind.

Of course, we could upgrade the mqtt-client version. Thanks for pointing
this out! I will do this.

Thanks

On Fri, Nov 14, 2014 at 6:19 AM, Jeffrey Nguyen (jeffrngu) <
jeffrngu@cisco.com> wrote:

>  Hi,
>
>  While testing Stratos 4.1 M3, I noticed about half the times,
> unsubscribing a cartridge takes 60+ seconds to execute.  During that time,
> the caller is blocked.  I poke around Stratos code and found TopicPublisher
> class that is responsible for the 60 second retry delay.  The code sleeps
> for 60 seconds whenever we encounter an exception while posting the given
> message to its corresponding topic.   Looks like this code has been around
> since at least release 4.0.0.
>
>  It seems to me 60 sec delay is way to long.  If you have a lot of
> subscriptions, the delay can be multiple factor of 60 seconds.   Also, if
> you do this via Stratos' Rest API, the request might get timed out before
> the response comes back?
>
>  I reduced this delay from 60 seconds to one second and tested and
> noticed that worked just fine.   I'm planning to push this change upstream
> unless I get any objection on that.
>
>  As for why the exception while attempting to update the topic, I did
> some googling and found [1].   It seems this is an issue with mqtt-client.
>   Somehow we're getting a stale connection to MB when we update the topic.
>   We're currently using version 0.4.0 of mqtt-client.  From [2], it looks
> like the latest version is 1.0.0.   Maybe it's time to upgrade?
>
>  Other question: when I make a Rest call to get list of subscriptions and
> it returns nothing, is it safe to assume all spawn instances have been
> killed?
>
>  Regards,
> -Jeffrey
>
>  [1] https://github.com/openhab/openhab/issues/980
> *[2] *http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: Un-subscribe cartridge takes 60+ sec

Posted by Isuru Haththotuwa <is...@apache.org>.
Hi Jeffrey,

On Fri, Nov 14, 2014 at 6:19 AM, Jeffrey Nguyen (jeffrngu) <
jeffrngu@cisco.com> wrote:

>  Hi,
>
>  While testing Stratos 4.1 M3, I noticed about half the times,
> unsubscribing a cartridge takes 60+ seconds to execute.  During that time,
> the caller is blocked.  I poke around Stratos code and found TopicPublisher
> class that is responsible for the 60 second retry delay.  The code sleeps
> for 60 seconds whenever we encounter an exception while posting the given
> message to its corresponding topic.   Looks like this code has been around
> since at least release 4.0.0.
>
>  It seems to me 60 sec delay is way to long.  If you have a lot of
> subscriptions, the delay can be multiple factor of 60 seconds.   Also, if
> you do this via Stratos' Rest API, the request might get timed out before
> the response comes back?
>
>  I reduced this delay from 60 seconds to one second and tested and
> noticed that worked just fine.   I'm planning to push this change upstream
> unless I get any objection on that.
>
As Udara mentioned, we can improve this algorithm to start from 1 second
and then increase the time interval that publisher waits before trying to
publish. If the broker/connection is not available for a long time, no
point in keep trying per one second.

>
>  As for why the exception while attempting to update the topic, I did
> some googling and found [1].   It seems this is an issue with mqtt-client.
>   Somehow we're getting a stale connection to MB when we update the topic.
>   We're currently using version 0.4.0 of mqtt-client.  From [2], it looks
> like the latest version is 1.0.0.   Maybe it's time to upgrade?
>
+1

>
>  Other question: when I make a Rest call to get list of subscriptions and
> it returns nothing, is it safe to assume all spawn instances have been
> killed?
>
>  Regards,
> -Jeffrey
>
>  [1] https://github.com/openhab/openhab/issues/980
> *[2] *http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/
>
> --
> <http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/>
> <http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/>
> Thanks and Regards,
>
> Isuru H.
> <http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/>
> +94 716 358 048
> <http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/>*
> <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: Un-subscribe cartridge takes 60+ sec

Posted by Udara Liyanage <ud...@wso2.com>.
Hi,

Yes 60 seconds is a long time. We can start from short value and increase
timeout when failed as Imesh mentioned once.



Touched, not typed. Erroneous words are a feature, not a typo.
On Nov 14, 2014 6:19 AM, "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>
wrote:

>  Hi,
>
>  While testing Stratos 4.1 M3, I noticed about half the times,
> unsubscribing a cartridge takes 60+ seconds to execute.  During that time,
> the caller is blocked.  I poke around Stratos code and found TopicPublisher
> class that is responsible for the 60 second retry delay.  The code sleeps
> for 60 seconds whenever we encounter an exception while posting the given
> message to its corresponding topic.   Looks like this code has been around
> since at least release 4.0.0.
>
>  It seems to me 60 sec delay is way to long.  If you have a lot of
> subscriptions, the delay can be multiple factor of 60 seconds.   Also, if
> you do this via Stratos' Rest API, the request might get timed out before
> the response comes back?
>
>  I reduced this delay from 60 seconds to one second and tested and
> noticed that worked just fine.   I'm planning to push this change upstream
> unless I get any objection on that.
>
>  As for why the exception while attempting to update the topic, I did
> some googling and found [1].   It seems this is an issue with mqtt-client.
>   Somehow we're getting a stale connection to MB when we update the topic.
>   We're currently using version 0.4.0 of mqtt-client.  From [2], it looks
> like the latest version is 1.0.0.   Maybe it's time to upgrade?
>
>  Other question: when I make a Rest call to get list of subscriptions and
> it returns nothing, is it safe to assume all spawn instances have been
> killed?
>
>  Regards,
> -Jeffrey
>
>  [1] https://github.com/openhab/openhab/issues/980
> *[2] *http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/
>