You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shindig.apache.org by "Ciancetta, Jesse E." <jc...@mitre.org> on 2012/04/25 13:44:09 UTC
Shindig HttpFetcher
Hi All,
We're running into an issue with our HttpFetcher implementation where slow responses to outgoing HTTP requests are crashing our shindig server. Every once in a great while one of the services our gadgets call on for data will have some kind of issue and start returning really slowly which causes a backup of requests in the fetcher to the point that shindig starts to become unresponsive.
So we're basically looking to implement some kind of dynamic throttling in our fetcher where we can detect that some service is having a problem and throttle down requests to that service until its response times return back to normal. However, one detail that makes our case a little trickier that normal (or maybe not depending on how common this practice is) is that we mask many different services behind a single host using a reverse proxy -- so all of these URL's on the same host may actually be hitting different servers on the backend:
host.example.com/some-app
host.example.com/some-collection-of-similar-apps/app-1/api/foo
host.example.com/some-collection-of-similar-apps/app-1/api/foo/bar
host.example.com/some-collection-of-similar-apps/app-1/api/foo/...
host.example.com/some-collection-of-similar-apps/app-2
host.example.com/some-collection-of-similar-apps/app-2/foo
host.example.com/different-app
...
That makes truly dynamic throttling kind of tricky since the paths can vary arbitrarily -- so for example if this app was the one that was slow:
host.example.com/some-collection-of-similar-apps/app-2
we want to throttle down requests to just that one app but let the others continue through.
One other detail -- some of the apps behind our reverse proxy are legacy apps which are obnoxiously slow so we have to set the timeout on our fetcher to an obnoxiously high value to accommodate those legacy services which makes this problem just that much worse...
The only solution I've come up with so far is to define some sort of regular expressions or something that we can use to figure out what app a given URL belongs to and then using that we can see that app-2 is slow and start throttling requests to it -- but that's a very manual/specific solution to the problem and we were hoping to come up with something more generic that we could push back into shindig core.
Has anyone else has faced similar issues and/or have any suggestions on how to go about handling this?
Thanks!
--Jesse
RE: Shindig HttpFetcher
Posted by "Ciancetta, Jesse E." <jc...@mitre.org>.
Thanks Stanton -- more comments inline.
>-----Original Message-----
>From: Stanton Sievers [mailto:ssievers@us.ibm.com]
>Sent: Thursday, April 26, 2012 1:19 PM
>To: dev@shindig.apache.org
>Subject: Re: Shindig HttpFetcher
>
>Hi Jesse,
>
>I'm looking at BasicHttpFetcher and it seems like someone had this idea
>(or something similar) in mind when implementing it. There's logic in the
>fetch method to check for slow responses and call a protected
>slowResponseWarning method that currently only logs a warning if a
>response took too long. The comments on slowResponseWarning seem to
>suggest overriding it "if you want to do something other than logging a
>warning". From an implementation standpoint, you could keep track of slow
>responses via this method and then override the fetch method to check if
>the incoming request has been flagged as slow before calling super.fetch.
>
>This still doesn't answer your question about how to best figure out what
>apps are slow when using a reverse proxy,
Right -- that's the piece that I'm not sure how best to handle... I appreciate your response though!
> but I think that's an issue that
>doesn't need to be solved in Shindig. It's pretty much implementation
>specific at that point.
I was thinking if we could come up with a generic enough solution it might be useful for many/most implementers. If we come up with something generic enough I'll put a review out and we can decide if it's useful/generic enough for shindig core.
>I don't think I really answered your question, but figured I would throw
>in my 2 cents anyways. :)
>
>Thanks,
>-Stanton
>
>
>
>From: "Ciancetta, Jesse E." <jc...@mitre.org>
>To: "dev@shindig.apache.org" <de...@shindig.apache.org>,
>Date: 04/25/2012 07:44
>Subject: Shindig HttpFetcher
>
>
>
>Hi All,
>
>We're running into an issue with our HttpFetcher implementation where slow
>responses to outgoing HTTP requests are crashing our shindig server. Every
>once in a great while one of the services our gadgets call on for data
>will have some kind of issue and start returning really slowly which
>causes a backup of requests in the fetcher to the point that shindig
>starts to become unresponsive.
>
>So we're basically looking to implement some kind of dynamic throttling in
>our fetcher where we can detect that some service is having a problem and
>throttle down requests to that service until its response times return
>back to normal. However, one detail that makes our case a little trickier
>that normal (or maybe not depending on how common this practice is) is
>that we mask many different services behind a single host using a reverse
>proxy -- so all of these URL's on the same host may actually be hitting
>different servers on the backend:
>
>host.example.com/some-app
>host.example.com/some-collection-of-similar-apps/app-1/api/foo
>host.example.com/some-collection-of-similar-apps/app-1/api/foo/bar
>host.example.com/some-collection-of-similar-apps/app-1/api/foo/...
>host.example.com/some-collection-of-similar-apps/app-2
>host.example.com/some-collection-of-similar-apps/app-2/foo
>host.example.com/different-app
>...
>
>That makes truly dynamic throttling kind of tricky since the paths can
>vary arbitrarily -- so for example if this app was the one that was slow:
>
>host.example.com/some-collection-of-similar-apps/app-2
>
>we want to throttle down requests to just that one app but let the others
>continue through.
>
>One other detail -- some of the apps behind our reverse proxy are legacy
>apps which are obnoxiously slow so we have to set the timeout on our
>fetcher to an obnoxiously high value to accommodate those legacy services
>which makes this problem just that much worse...
>
>The only solution I've come up with so far is to define some sort of
>regular expressions or something that we can use to figure out what app a
>given URL belongs to and then using that we can see that app-2 is slow and
>start throttling requests to it -- but that's a very manual/specific
>solution to the problem and we were hoping to come up with something
>more
>generic that we could push back into shindig core.
>
>Has anyone else has faced similar issues and/or have any suggestions on
>how to go about handling this?
>
>Thanks!
>
>--Jesse
>
Re: Shindig HttpFetcher
Posted by Stanton Sievers <ss...@us.ibm.com>.
Hi Jesse,
I'm looking at BasicHttpFetcher and it seems like someone had this idea
(or something similar) in mind when implementing it. There's logic in the
fetch method to check for slow responses and call a protected
slowResponseWarning method that currently only logs a warning if a
response took too long. The comments on slowResponseWarning seem to
suggest overriding it "if you want to do something other than logging a
warning". From an implementation standpoint, you could keep track of slow
responses via this method and then override the fetch method to check if
the incoming request has been flagged as slow before calling super.fetch.
This still doesn't answer your question about how to best figure out what
apps are slow when using a reverse proxy, but I think that's an issue that
doesn't need to be solved in Shindig. It's pretty much implementation
specific at that point.
I don't think I really answered your question, but figured I would throw
in my 2 cents anyways. :)
Thanks,
-Stanton
From: "Ciancetta, Jesse E." <jc...@mitre.org>
To: "dev@shindig.apache.org" <de...@shindig.apache.org>,
Date: 04/25/2012 07:44
Subject: Shindig HttpFetcher
Hi All,
We're running into an issue with our HttpFetcher implementation where slow
responses to outgoing HTTP requests are crashing our shindig server. Every
once in a great while one of the services our gadgets call on for data
will have some kind of issue and start returning really slowly which
causes a backup of requests in the fetcher to the point that shindig
starts to become unresponsive.
So we're basically looking to implement some kind of dynamic throttling in
our fetcher where we can detect that some service is having a problem and
throttle down requests to that service until its response times return
back to normal. However, one detail that makes our case a little trickier
that normal (or maybe not depending on how common this practice is) is
that we mask many different services behind a single host using a reverse
proxy -- so all of these URL's on the same host may actually be hitting
different servers on the backend:
host.example.com/some-app
host.example.com/some-collection-of-similar-apps/app-1/api/foo
host.example.com/some-collection-of-similar-apps/app-1/api/foo/bar
host.example.com/some-collection-of-similar-apps/app-1/api/foo/...
host.example.com/some-collection-of-similar-apps/app-2
host.example.com/some-collection-of-similar-apps/app-2/foo
host.example.com/different-app
...
That makes truly dynamic throttling kind of tricky since the paths can
vary arbitrarily -- so for example if this app was the one that was slow:
host.example.com/some-collection-of-similar-apps/app-2
we want to throttle down requests to just that one app but let the others
continue through.
One other detail -- some of the apps behind our reverse proxy are legacy
apps which are obnoxiously slow so we have to set the timeout on our
fetcher to an obnoxiously high value to accommodate those legacy services
which makes this problem just that much worse...
The only solution I've come up with so far is to define some sort of
regular expressions or something that we can use to figure out what app a
given URL belongs to and then using that we can see that app-2 is slow and
start throttling requests to it -- but that's a very manual/specific
solution to the problem and we were hoping to come up with something more
generic that we could push back into shindig core.
Has anyone else has faced similar issues and/or have any suggestions on
how to go about handling this?
Thanks!
--Jesse