You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Daniel Ruggeri <DR...@primary.net> on 2015/05/01 02:30:19 UTC
Re: Proposal/RFC: "informed" load balancing
On 4/29/2015 11:54 PM, Jim Riggs wrote:
> [ Long message and proposal follows. Bear with me. There are a lot of words, but that is because we need a lot of help/input! ;-) ]
>
> So, this has come up in the past several times, and we discussed it again this year at ApacheCon: How do we get the load balancer to make smarter, more informed decisions about where to send traffic?
>
> The different LB methods provide some different attempts at balancing traffic, but ultimately none of them is "smart" about its decision. Other than a member being in error state, the balancer makes its decision solely based on configuration (LB set, factor, etc.) and its own knowledge of the member (e.g. requests, bytes). What we have often discussed is a way to get some type of health/load/capacity information from the backend to make informed balancing decisions.
>
> One method is to use health checks (a la haproxy, AWS ELBs, etc.) that request one or more URLs and the response code/time indicates whether or not the service is up and available, allowing more proactive decisions. While this is better than our current state of reactively marking members in error state based on failed requests, it still provides a limited view of the health/state of the backend.
>
> We have also discussed implementing a way for backends to communicate a magical "load" number to the front end to take into account as it balances traffic. This would give a much better view into the backend's state, but requires some way to come up with this calculation that each backend system/server/service/app must provide. This then has to be implemented in all the various backends (e.g. httpd, tomcat, php-fpm, unicorn, mongrel, etc., etc.), probably a hard sell to all of those projects. And, the front end would have limited control over what that number means or how to use it.
>
> During JimJag's balancer talk at ApacheCon this year, he brought up this issue of "better, more informed" decision making again. I put some thought into it that night and came up with some ideas. Jim, Covener, Trawick, Ruggeri, and I then spent some time over the next couple of days talking it through and fleshing out some of the details.
>
> Based on all of that, below is what I am proposing. I have some initial code that I am working on to implement the different pieces of this, and I will put them up in bugz or somewhere when they're a little less rudimentary.
>
> --
>
> Our hope is to create a general standard that can be used by various projects, products, proxies, servers, etc., to have a more consistent way for a load balancer to request and receive useful internal state information from its backend nodes. This information can then be used by the *frontend* software/admin (this is the main change from what we have discussed before) to calculate a load factor appropriate for each backend node.
>
> This communication uses a new, standard HTTP header, "X-Backend-Info", that takes this form in RFC2616 BNF:
>
> backend-info = "version" "=" numeric-entry
> [
> *LWS "," *LWS
> #( numeric-entry | string-entry )
> ]
>
> numeric-entry = numeric-field "=" ( float | <"> float <"> )
> ; that is, numbers may optionally be enclosed in
> ; quotation marks
>
> float = 1*DIGIT [ "." 1*DIGIT ]
>
> numeric-field = "workers-max"
> ; maximum number of workers the backend supports
> | "workers-used"
> ; current number of used/busy workers
> | "workers-allocated"
> ; current number of allocated/ready workers
> | "workers-free"
> ; current number of workers available for use
> ; (generally the difference between workers-max and
> ; workers-used, though some implementations may have
> ; a different notion)
> | "uptime"
> ; number of seconds the backend has been running
> | "requests"
> ; number of requests the backend has processed
> | "memory-max"
> ; total amount of memory available in bytes
> | "memory-used"
> ; amount of used memory in bytes
> | "memory-allocated"
> ; amount of allocated/committed memory in bytes
> | "memory-free"
> ; amount of memory available for use (generally
> ; the difference between memory-max and memory-used,
> ; though some implementations may have a different
> ; notion)
> | "load-current"
> ; the (subjective) current load for the backend
> | "load-5"
> ; the (subjective) 5-minute load for the backend
> | "load-15"
> ; the (subjective) 15-minute load for the backend
>
> string-entry = string-field "=" ( token | quoted-string )
A useful token could be status=OK|ERROR|MAINTENANCE where a backend
could advertise to the upstream load balancer that it may want to be put
in drain mode or something to that effect.
Since this list can't/won't be exhaustive of all things people could
care to send, let's add some head room in the spec by allowing
"custom-integer" and/or "custom-string". Otherwise, I suspect people
would cram things into the wrong fields just to get the data back to the
proxy.
> string-field = "provider"
> ; informational description of backend information
> ; provider (module, container, subsystem, app, etc.)
>
>
> As used here, "worker" is an overloaded term whose precise meaning is backend-dependent. It might refer to processes, threads, pipelines, or whatever the backend system/server/service/app uses to measure or limit its number of active, processing connections.
>
> The process-flow looks like this:
>
> 1. The frontend (periodically based on time or requests, or on demand) as part of either (1) a normal proxied request or (2) a dedicated health check adds an "X-Backend-Info" request header to a backend request, informing the backend that it wants node state information. I.e.:
>
> X-Backend-Info: version=1.0
>
> 2. The backend node receives a request with an "X-Backend-Info" header specifying a version it supports.
>
> 3. A supporting backend node SHOULD insert one or more "X-Backend-Info" response headers with any subset of the backend-info fields that it supports, including the required "version" field. The version of information provided MUST be less than or equal to the version requested. (The fields are standardized so that various frontends know what to expect, rather than each backend system/server/service/app creating its own fields/values.) E.g.:
>
> X-Backend-Info: version=1.0, provider="Backend X", workers-max=1000,
> workers-used=517, workers-free=483, uptime=19234,
> requests=85939
I wonder what the practical applications could be for the "or more" case
- it seems like the intent is for different pools or resources to be
returned. If so, see below...
>
> 4. The backend MUST add the "X-Backend-Info" token to the "Connection" response header, making it a hop-by-hop field that is removed by the frontend from the downstream response (RFC2616 14.10 and RFC7230 6.1). [Note there appears to be an httpd bug here that I intend to submit and that needs to be addressed.]
>
> Connection: X-Backend-Info
I'm not sure if this is a stroke of brilliance or extra work that isn't
needed :-) . As we discussed at the Con, it is vital for the proxy to
remove the header to avoid leaking any potentially useful information to
an attacker out to the 'tubes... but parsing Connection for
"X-Backend-Info" seems like it wouldn't be needed since one could just
as well check if X-Backend-Info header is present. I'm probably missing
the obvious, but can you help me understand more about why we would want
this here instead of treating the presence of the header as a sign to do
some kind of work?
>
> 5. The frontend parses the backend-info entries in the received "X-Backend-Info" response header. The values are then used as part of either an internal or an administrator-specified calculation to determine the load factor or weight of that node for subsequent requests.
Or taken as Gospel Truth and applied directly after some basic validate?
> 6. The frontend MUST remove the "X-Backend-Info" hop-to-hop response header per RFCs.
>
> --
>
> As for httpd implementation, this has two pieces. The first is when httpd is used as a backend node behind a load balancer and must provide X-Backend-Info response data. For this, I have created a module tentatively named mod_proxy_backend_info that does nothing except insert an output filter to populate the response header with version, provider, workers-*, request, uptime, and load-* values when the request header is present. Here is an example request-response:
>
> % curl -v -H 'X-Backend-Info: version=1.0' http://localhost/
> * Trying 127.0.0.1...
> * Connected to localhost (127.0.0.1) port 80 (#0)
>> GET / HTTP/1.1
>> User-Agent: curl/7.41.0
>> Host: localhost
>> Accept: */*
>> X-Backend-Info: version=1.0
>>
> < HTTP/1.1 200 OK
> < Date: Thu, 30 Apr 2015 04:32:08 GMT
> < Server: Apache/2.4.9 (Unix) PHP/5.5.14
> < Last-Modified: Wed, 15 Apr 2015 14:04:54 GMT
> < ETag: "2d-513c3d4d78d80"
> < Accept-Ranges: bytes
> < Content-Length: 45
> < X-Backend-Info: version=1.0, provider="mod_proxy_backend_info [Apache/2.4.9 (Unix) PHP/5.5.14]", workers-max=256, workers-busy=1, workers-ready=4, workers-free=255, uptime=1448, requests=3, load-current=1.737305, load-5=1.733887, load-15=1.668457
> < Connection: X-Backend-Info
> < Content-Type: text/html
> <
> <html><body><h1>It works!</h1></body></html>
>
>
> The second piece is when httpd is used as the load balancer. For this, I have created a module tentatively named mod_lbmethod_bybackendinfo that will:
>
> 1. Periodically (based on elapsed time, number of requests, or both since last update) insert the X-Backend-Info request header into a proxied request.
>
> 2. Parse and remove the X-Backend-Info response header.
>
> 3. Calculate the member's "informed" load factor based on a formula specified by the user/admin in the configuration. I hope to just use the existing lbfactor field to store this calculated value. Then we can use existing logic to balance based on lbset and lbfactor for subsequent requests.
>
> 4. Store the current time and request count in the member's data structure so the lbmethod knows when it needs to be updated again.
>
>
> What I need from all of you:
>
> - Input/commentary on the proposed idea, approach, and implementation. Renaming things, additional fields that might be useful, etc., are all up for discussion.
>
> - Help with handling the configuration formula mentioned in #3 above. Can we just add some math operators to the expression parser to handle this? What all operations/functions might we need (+-*/? max? min? ternary if-then-else? ...)? A simple-ish example (something like this maybe?):
>
> <Proxy "balancer://...">
> BalancerMember ...
> ...
> ProxySet \
> lbmethod=bybackendinfo \
> backendupdateseconds=30 \
> backendupdaterequests=100 \
> backendformula="%{BACKEND:uptime} -lt 120 ? 1 : %{BACKEND:workers-free} / %{BACKEND:workers-max} * 100"
> </Proxy>
Since the spec above states that one or more X-Backend-Info headers is
acceptable, there should be a way for the admin to declare upon which
header the calculation should be performed. A provider flag or option
(FIRST|LAST|"provider-name") ought to do the trick.
Side note: Is there a generic place for maths based on strings in httpd
at all? Creating our own mini-language makes me nervous since we have to
consider that backends can and will do stupid things that could result
in httpd attempting an operation that could cause a crash if we aren't
very careful about validating inputs and making sure we aren't doing
silly things like div/0
>
> - [Near-long-term] Help adding X-Backend-Info backend support and documentation to various projects. Tomcat, php-fpm, others(?) should be fairly easy to implement and submit patches. This work does us no good if none of our backends support it.
I will be happy to provide a ServletFilter that can be used in a J2EE
app or a Tomcat Valve (good for the many Tomcat variations and JBoss)
once we settle on details.
>
> - [Long-term] Help adding X-Backend-Info frontend support and documentation to various projects to help this become an "accepted ad-hoc standard"...or something like that. Nginx, haproxy, and many others would be targets.
>
>
> Warn out from writing all of this and hopeful that someone other than me actually cares, I wish you all well today/tonight!
>
> - Jim
>
All in all, man, this is solid. I like what you've done here.
--
Daniel Ruggeri
Re: Proposal/RFC: "informed" load balancing
Posted by Tim Bannister <is...@c8h10n4o2.org.uk>.
On 1 May 2015, at 01:30, Daniel Ruggeri <DR...@primary.net> wrote:
>>
>> 4. The backend MUST add the "X-Backend-Info" token to the "Connection" response header, making it a hop-by-hop field that is removed by the frontend from the downstream response (RFC2616 14.10 and RFC7230 6.1). [Note there appears to be an httpd bug here that I intend to submit and that needs to be addressed.]
>>
>> Connection: X-Backend-Info
>
> I'm not sure if this is a stroke of brilliance or extra work that isn't needed :-) . As we discussed at the Con, it is vital for the proxy to remove the header to avoid leaking any potentially useful information to an attacker out to the 'tubes... but parsing Connection for "X-Backend-Info" seems like it wouldn't be needed since one could just as well check if X-Backend-Info header is present. I'm probably missing the obvious, but can you help me understand more about why we would want this here instead of treating the presence of the header as a sign to do some kind of work?
Here's a situation that could go wrong if this new header weren't marked as hop-by-hop. Imagine if there are two webserver products in a reverse proxy topology, something like this:
user-agent ← httpd-proxy ← acme-proxy ← httpd-origin
(the server tiers might in fact be clusters of identically configured hosts).
All 4 tiers are doing HTTP/1.1 cacheing, correctly using Vary: and so on. If httpd-origin is sending X-Backend-Info then it must signal to ACME-proxy that this is a hop-by-hop header. Let's say httpd-origin signals that workers-free is 0. httpd-proxy receives a copy of this header and from acme-proxy. httpd-proxy incorrectly concludes that workers-free is 0 and starts sending 503 responses as per its intended configuration, even though acme-proxy would be able to serve stale responses from its cache.
The sysadmin contacts the vendor “ACME Proxy”; the vendor asserts that their product is conforming to HTTP 1.1 and that the incorrect behaviour is in Apache httpd. Which, in my view, it would be.
--
Tim Bannister – isoma@c8h10n4o2.org.uk
Re: Proposal/RFC: "informed" load balancing
Posted by Tim Bannister <is...@c8h10n4o2.org.uk>.
On 1 May 2015, at 01:30, Daniel Ruggeri <DR...@primary.net> wrote:
>
> On 4/29/2015 11:54 PM, Jim Riggs wrote:
>>
>> So, this has come up in the past several times, and we discussed it again this year at ApacheCon: How do we get the load balancer to make smarter, more informed decisions about where to send traffic?
…
>> string-entry = string-field "=" ( token | quoted-string )
>
> A useful token could be status=OK|ERROR|MAINTENANCE where a backend could advertise to the upstream load balancer that it may want to be put in drain mode or something to that effect. Since this list can't/won't be exhaustive of all things people could care to send, let's add some head room in the spec by allowing "custom-integer" and/or "custom-string". Otherwise, I suspect people would cram things into the wrong fields just to get the data back to the proxy.
Although I think it's an approach deprecated by IETF, how about allowing any field name provided it's prefixed with “x-”?
--
Tim Bannister – isoma@c8h10n4o2.org.uk