You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Yann Ylavic <yl...@gmail.com> on 2018/02/01 17:54:31 UTC
New ServerUID directive
I have this patch (attached) floating around that allows users to
configure a *fixed* UID for each vhost.
There are several places where we need an ID for vhosts, and where we
compute one based on ServerName/Port, addresses, configuration path
and line numbers, ...
This UID could be used instead, moreover since it has the property to
not change on (re)starts we can also use it for things bound to a
vhost regardless of the startup and unrelated configuration changes
(the attached patch uses it for SHMs in mod_proxy_balancer, as an
example).
If no ServerUID is configured, I _think_ we can compute one too,
better than the one usually computed in our code since it won't change
unless address(es)/port(s) or ServerName of the vhost changes (which
is not a "light" change anyway).
In any case the patch also handles collisions, if ever...
So for this what the patch does is (with rationale in comment):
+ int i, *num;
+ apr_hash_t *servers_uids = apr_hash_make(p);
[]
+ if (!s->server_uid) {
+ server_addr_rec *addr;
+ apr_md5_ctx_t md5_ctx;
+ unsigned char md5[APR_MD5_DIGESTSIZE];
+
+ /* Assumes the unique identifier of a vhost is its address(es)
+ * plus the ServerName:Port. Should two or more vhosts have this
+ * same identifier, the first one would always be elected to
+ * handle the requests, so this shouldn't be an issue...
+ */
+ apr_md5_init(&md5_ctx);
+ for (addr = s->addrs; addr; addr = addr->next) {
+ char host_ip[64]; /* for any IPv[46] string */
+ apr_sockaddr_ip_getbuf(host_ip, sizeof host_ip,
+ addr->host_addr);
+ apr_md5_update(&md5_ctx, (unsigned char *)host_ip,
+ strlen(host_ip));
+ apr_md5_update(&md5_ctx, (unsigned char *)&addr->host_port,
+ sizeof(addr->host_port));
+ }
+ apr_md5_update(&md5_ctx, (unsigned char *)s->server_hostname,
+ strlen(s->server_hostname));
+ apr_md5_update(&md5_ctx, (unsigned char *)&s->port,
+ sizeof(s->port));
+ apr_md5_final(md5, &md5_ctx);
+
+ s->server_uid = apr_pescape_hex(p, md5, sizeof md5, 0);
+ }
+ /* Handle collisions, that's Unique ID! */
+ num = apr_hash_get(servers_uids, s->server_uid,
+ APR_HASH_KEY_STRING);
+ if (num) {
+ ++*num;
+ s->server_uid = apr_psprintf(p, "%s_%i", s->server_uid, *num);
+ }
+ else {
+ num = apr_pcalloc(p, sizeof *num);
+ apr_hash_set(servers_uids, s->server_uid,
+ APR_HASH_KEY_STRING, num);
+ }
(Rest attached)
WDYT?
Re: New ServerUID directive
Posted by Michael <ai...@felt.demon.nl>.
On 06/02/2018 11:54, Stefan Eissing wrote:
>
>> Am 06.02.2018 um 11:45 schrieb Helmut K. C. Tessarek <te...@evermeet.cx>:
>>
>> On 2018-02-06 05:13, Yann Ylavic wrote:
>>> Sorry for what is probably (my) bad english, "fixed" meant "the same
>>> after restart (or stop/start)".
>> Right, but isn't the virtual host's server name/port config after the
>> restart the same as well? Why do you need a new separate unique identifier?
>>
>> And should you ever change the port number and/or the virtual host's
>> server name, then this virtual host won't be the same after a restart
>> anyway.
>>
>> Either I'm missing something here, but I still don't understand the
>> reason for a unique identifier, when you already have one.
> You are missing that Yann exactly wants to do that.
>
> Only as consideration for people who prefer otherwise, he considered to
> introduce a ServerUID directive.
>
> Now, he tried several times to get the discussion back to what a good
> *automatic* id for the load balancer is,
Ah, for the fortunate that have so much traffic they need the 'lb'. And
I imagine, for that 'automatic' is fine. Never had to use one though -
so no idea how hard they are to configure/manage. However, I expect I
would rather "not care" how the internals work for giving me a vhost
ServerID. Why should I care - after a restart whether the value
generated is the same or not.
That said - what could I do with a ServerID (forget the unique for the
moment).
Again, my first thoughts are with regard to 'security' aka 'access
control'. Could I use (or is there already something I am unaware of) a
ServerID in <Directory> blocks, e.g., with <Require> - so that I can
specify access control in terms of the <vhost> rather than as attributes
of clients. Might all be nonsense - asin - this is just me brainstorming.
I guess my question is closer to: are there ways to manage 'access
control' based on the server configuration and the physical resources
(mainly thinking files). What is more manageable? What is easier to
report on/with (to a non-httpd specialist). What is easier to audit/log,
perhaps in separate logs?
> but everyone keeps discussing
> directives...
>
> *Waves Jedi Hand*: "Forget the directive..."
(* Michael blinks - what were we talking about? *)
>
>> Or at least one that can be used from a combination of several fields in
>> the server struct.
>>
>> What am I missing?
>>
>> --
>> regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
>> Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944
>>
>> /*
>> Thou shalt not follow the NULL pointer for chaos and madness
>> await thee at its end.
>> */
>>
>
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Tue, Feb 6, 2018 at 11:54 AM, Stefan Eissing
<st...@greenbytes.de> wrote:
>
>> Am 06.02.2018 um 11:45 schrieb Helmut K. C. Tessarek <te...@evermeet.cx>:
>>
>> On 2018-02-06 05:13, Yann Ylavic wrote:
>>> Sorry for what is probably (my) bad english, "fixed" meant "the same
>>> after restart (or stop/start)".
>>
>> And should you ever change the port number and/or the virtual host's
>> server name, then this virtual host won't be the same after a restart
>> anyway.
Which may be irrelevent for, say, a module which cares only about
backend connections.
Why would adding another incoming IP:port or ServerAlias matter there?
>>
>> Either I'm missing something here, but I still don't understand the
>> reason for a unique identifier, when you already have one.
>
> You are missing that Yann exactly wants to do that.
>
> Only as consideration for people who prefer otherwise, he considered to
> introduce a ServerUID directive.
>
> Now, he tried several times to get the discussion back to what a good
> *automatic* id for the load balancer is, but everyone keeps discussing
> directives...
Thanks Stefan, I couldn't have said better!
>
> *Waves Jedi Hand*: "Forget the directive..."
May the force be with you ;)
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
But we already HAVE an "automatic" ID... that's what the Vhost's
server_rec is. The issue is that there is no "one-size-fits-all" definition
of what such an automatic ID means or how it should be defined.
Is it just Vhost name + Port? That might be good for some modules,
but NOT good for others. What if the definition/config of the
Vhost changes between restarts? Should the 2 Vhosts have
the same ID (ie: are they the "same" Vhosts?) Again, for some
modules maybe yes, others maybe no. It is up to each module
to know and use what makes sense for itself.
An automatic ID is just unnecessary fluff... It's an API data field
that provides no real useful information beyond that which we
already provide.
> On Feb 6, 2018, at 5:54 AM, Stefan Eissing <st...@greenbytes.de> wrote:
>
>
>
>> Am 06.02.2018 um 11:45 schrieb Helmut K. C. Tessarek <te...@evermeet.cx>:
>>
>> On 2018-02-06 05:13, Yann Ylavic wrote:
>>> Sorry for what is probably (my) bad english, "fixed" meant "the same
>>> after restart (or stop/start)".
>>
>> Right, but isn't the virtual host's server name/port config after the
>> restart the same as well? Why do you need a new separate unique identifier?
>>
>> And should you ever change the port number and/or the virtual host's
>> server name, then this virtual host won't be the same after a restart
>> anyway.
>>
>> Either I'm missing something here, but I still don't understand the
>> reason for a unique identifier, when you already have one.
>
> You are missing that Yann exactly wants to do that.
>
> Only as consideration for people who prefer otherwise, he considered to
> introduce a ServerUID directive.
>
> Now, he tried several times to get the discussion back to what a good
> *automatic* id for the load balancer is, but everyone keeps discussing
> directives...
>
> *Waves Jedi Hand*: "Forget the directive..."
>
>>
>> Or at least one that can be used from a combination of several fields in
>> the server struct.
>>
>> What am I missing?
>>
>> --
>> regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
>> Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944
>>
>> /*
>> Thou shalt not follow the NULL pointer for chaos and madness
>> await thee at its end.
>> */
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 06.02.2018 um 11:45 schrieb Helmut K. C. Tessarek <te...@evermeet.cx>:
>
> On 2018-02-06 05:13, Yann Ylavic wrote:
>> Sorry for what is probably (my) bad english, "fixed" meant "the same
>> after restart (or stop/start)".
>
> Right, but isn't the virtual host's server name/port config after the
> restart the same as well? Why do you need a new separate unique identifier?
>
> And should you ever change the port number and/or the virtual host's
> server name, then this virtual host won't be the same after a restart
> anyway.
>
> Either I'm missing something here, but I still don't understand the
> reason for a unique identifier, when you already have one.
You are missing that Yann exactly wants to do that.
Only as consideration for people who prefer otherwise, he considered to
introduce a ServerUID directive.
Now, he tried several times to get the discussion back to what a good
*automatic* id for the load balancer is, but everyone keeps discussing
directives...
*Waves Jedi Hand*: "Forget the directive..."
>
> Or at least one that can be used from a combination of several fields in
> the server struct.
>
> What am I missing?
>
> --
> regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
> Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944
>
> /*
> Thou shalt not follow the NULL pointer for chaos and madness
> await thee at its end.
> */
>
Re: New ServerUID directive
Posted by "Helmut K. C. Tessarek" <te...@evermeet.cx>.
On 2018-02-06 05:13, Yann Ylavic wrote:
> Sorry for what is probably (my) bad english, "fixed" meant "the same
> after restart (or stop/start)".
Right, but isn't the virtual host's server name/port config after the
restart the same as well? Why do you need a new separate unique identifier?
And should you ever change the port number and/or the virtual host's
server name, then this virtual host won't be the same after a restart
anyway.
Either I'm missing something here, but I still don't understand the
reason for a unique identifier, when you already have one.
Or at least one that can be used from a combination of several fields in
the server struct.
What am I missing?
--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944
/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Tue, Feb 6, 2018 at 11:27 AM, Michael <ai...@felt.demon.nl> wrote:
> On 06/02/2018 11:13, Yann Ylavic wrote:
>>
>> And the U in UID meant Unique, not User :/
>
> Then I think UUID - except you want it remembered over restarts - rather
> than for a session.
The first U there is Universal (Univeral Unique ID), and I didn't want
to impose a format either...
Simple ID would have done it probably (implies unique anyway).
>
> However, if it is not for 'external use' - how does it help me? What does it
> solve (ignorance is bliss!)? I got rather lost during the internal focused
> discussion.
It's both for "external" (e.g. %{SERVER_ID} anywhere you want) and
internal use where appropriate since this ID is garanteed to be unique
for the lifetime of the vhost (per admin choice), it could be used to
associate resources with the same (per admin choice) lifetime.
We can do things to compute this ID automatically when none is given,
but I think we can't do so *and* please everyone (doing configuration
changes), hence this long discussion...
Re: New ServerUID directive
Posted by Michael <ai...@felt.demon.nl>.
On 06/02/2018 11:13, Yann Ylavic wrote:
> On Tue, Feb 6, 2018 at 10:21 AM, Michael <ai...@felt.demon.nl> wrote:
>> On 01/02/2018 18:54, Yann Ylavic wrote:
>>
>> I have this patch (attached) floating around that allows users to
>> configure a *fixed* UID for each vhost.
>>
>> When I read the above sentence - with the emphasis on *fixed* my first
>> thoughts were not on httpd internals on how a (semi-)fixed ID is used
>> internally.
> Sorry for what is probably (my) bad english, "fixed" meant "the same
> after restart (or stop/start)".
I 'translated' it to mean - 'settable' (as a Directive) - and permanence
(restarts aka multiple sessions) and bound to the vhost.
>
>> Falls in the category of "don't want to know. Instead, my
>> initial reaction - this might be useful for having multiple UID - aka
>> "UserNames" or other sort of "external", UID.
> And the U in UID meant Unique, not User :/
Then I think UUID - except you want it remembered over restarts - rather
than for a session.
I 'strongly dislike' naming things (i.e., I have a terrible coming up
with names I like). PID (Permanant) ID is also confusing. Maybe VHID
(Virtual Host ID), or UVID or longer UVHID).
However, if it is not for 'external use' - how does it help me? What
does it solve (ignorance is bliss!)? I got rather lost during the
internal focused discussion.
>
> So I may not have choosen the right terms in the first place...
Maybe - but great new ideas come from confusion about what was initially
meant :)
>
>
> Regards,
> Yann.
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Tue, Feb 6, 2018 at 10:21 AM, Michael <ai...@felt.demon.nl> wrote:
> On 01/02/2018 18:54, Yann Ylavic wrote:
>
> I have this patch (attached) floating around that allows users to
> configure a *fixed* UID for each vhost.
>
> When I read the above sentence - with the emphasis on *fixed* my first
> thoughts were not on httpd internals on how a (semi-)fixed ID is used
> internally.
Sorry for what is probably (my) bad english, "fixed" meant "the same
after restart (or stop/start)".
> Falls in the category of "don't want to know. Instead, my
> initial reaction - this might be useful for having multiple UID - aka
> "UserNames" or other sort of "external", UID.
And the U in UID meant Unique, not User :/
So I may not have choosen the right terms in the first place...
Regards,
Yann.
Re: New ServerUID directive
Posted by Michael <ai...@felt.demon.nl>.
On 01/02/2018 18:54, Yann Ylavic wrote:
> I have this patch (attached) floating around that allows users to
> configure a*fixed* UID for each vhost.
I am not an httpd expert. And I probably already know more than I want to.
When I read the above sentence - with the emphasis on *fixed* my first
thoughts were not on httpd internals on how a (semi-)fixed ID is used
internally. Falls in the category of "don't want to know. Instead, my
initial reaction - this might be useful for having multiple UID - aka
"UserNames" or other sort of "external", UID.
As I read the discussions I realized my first thought was off - but
still I continued thinking about - are there ways to make a vhost UID
external, e.g., add to my logs for accountability. And I found myself
"dreaming" - how about a different UID (aka Username) that would
setuid() per vhost. Could be a nice way to separate data permissions -
per vhost.
So, maybe you do not really need it for something internal - per your
own discussion.
However, as a new "Directive" and all - what are ways this could be
applied to improve/enhance security and/or accountability. Is, perhaps,
the concept of a ServerUID as a new directive something that could be
useful to a complex website - and much more than config-file fluff?
I hope this helps - and is "out of the box" thinking. If not, well I
tried. :)
Good day all!
Michael
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Feb 2, 2018, at 9:42 AM, Stefan Eissing <st...@greenbytes.de> wrote:
>
>> Am 02.02.2018 um 15:25 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>>
>>
>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Jim Jagielski [mailto:jim@jaguNET.com]
>>> Gesendet: Freitag, 2. Februar 2018 15:15
>>> An: httpd <de...@httpd.apache.org>
>>> Betreff: Re: New ServerUID directive
>>>
>>> Why? If it is designed to not change between restarts then
>>> there are much easier ways to be unique, which it kinda
>>> already is, considering the actual structs being used.
>>>
>>> Also, this seems like unnecessary admin overhead for the
>>> webmaster... if there is a need for such an ID, httpd should
>>> provide for it automagically and not require users to have
>>> to config one. It seems like config-file fluff to me, IMO.
>>
>> +1. If the current ID used is not meeting our requirements, my first thought would be if we could improve it
>> to meet them.
>
> Totally agree that inventing a new Directive is the last resort if we cannot find a better solution.
>
> If I understand then concrete case here correctly, the admin makes frequent config changes and *wants* the shared memory id to stay the same, so the shm is re-used and not cleaned up or re-initialized. And our current implementation generates new identifiers much to frequently bc the line-number cheat was made to detect config changes?
I think that is correct, which implies, to me, the solution would
be some mod_proxy_balancer Directive like "IgnoreConfigFileChanges"
or something specific like that.
But I am still confused if the above is actually what the issue
is, hence my "resistance" for any modifications at all until
the problem is crystal clear. If we can check, in a one-off
test, that removing that line-number hack solves the "problem"
then that is more easily handled by a specific use-case for
balancer/slotmem.
My 2c
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 02.02.2018 um 15:25 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jim Jagielski [mailto:jim@jaguNET.com]
>> Gesendet: Freitag, 2. Februar 2018 15:15
>> An: httpd <de...@httpd.apache.org>
>> Betreff: Re: New ServerUID directive
>>
>> Why? If it is designed to not change between restarts then
>> there are much easier ways to be unique, which it kinda
>> already is, considering the actual structs being used.
>>
>> Also, this seems like unnecessary admin overhead for the
>> webmaster... if there is a need for such an ID, httpd should
>> provide for it automagically and not require users to have
>> to config one. It seems like config-file fluff to me, IMO.
>
> +1. If the current ID used is not meeting our requirements, my first thought would be if we could improve it
> to meet them.
Totally agree that inventing a new Directive is the last resort if we cannot find a better solution.
If I understand then concrete case here correctly, the admin makes frequent config changes and *wants* the shared memory id to stay the same, so the shm is re-used and not cleaned up or re-initialized. And our current implementation generates new identifiers much to frequently bc the line-number cheat was made to detect config changes?
-Stefan
Re: The Case for Managed Domains
Posted by Steffen <in...@apachelounge.com>.
I was triggered by IO stats he was mentioned:
For example,settings and locations for access logging would be nice to have. IO Stats
as well.
> Op 5 feb. 2018 om 02:47 heeft Eric Covener <co...@gmail.com> het volgende geschreven:
>
>> On Sun, Feb 4, 2018 at 8:43 PM, Luca Toscano <to...@gmail.com> wrote:
>> I am a bit ignorant about mod_bmx so I'd need to ask some follow up
>> questions: how is it going to solve Stefan's points? As a far as I can see
>> it is a very nice architecture to query metrics and process them via
>> plugins, so I'd be super glad to see it integrated to power up our status
>> page but I don't see how it could help in abstracting properties across
>> vhosts for example. Can anybody give me more info?
>
> FWIW I don't see the relationship either.
Re: The Case for Managed Domains
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Sun, Feb 4, 2018 at 7:47 PM, Eric Covener <co...@gmail.com> wrote:
> On Sun, Feb 4, 2018 at 8:43 PM, Luca Toscano <to...@gmail.com> wrote:
>> I am a bit ignorant about mod_bmx so I'd need to ask some follow up
>> questions: how is it going to solve Stefan's points? As a far as I can see
>> it is a very nice architecture to query metrics and process them via
>> plugins, so I'd be super glad to see it integrated to power up our status
>> page but I don't see how it could help in abstracting properties across
>> vhosts for example. Can anybody give me more info?
>
> FWIW I don't see the relationship either.
I don't see bmx serving the managed domain structure proposed in top post.
But the proposal could certainly alter the structure of bmx itself. The feature
I think Steffen refers to is mod_bmx_vhost, which collects traffic and activity
counts on a per-vhost basis.
In looking at the proposed model, if several vhosts reflected the same basic
unit of web services, I can see it evolving to count and sum the activity
across this management unit, as opposed to each individual vhost.
Re: The Case for Managed Domains
Posted by Eric Covener <co...@gmail.com>.
On Sun, Feb 4, 2018 at 8:43 PM, Luca Toscano <to...@gmail.com> wrote:
> I am a bit ignorant about mod_bmx so I'd need to ask some follow up
> questions: how is it going to solve Stefan's points? As a far as I can see
> it is a very nice architecture to query metrics and process them via
> plugins, so I'd be super glad to see it integrated to power up our status
> page but I don't see how it could help in abstracting properties across
> vhosts for example. Can anybody give me more info?
FWIW I don't see the relationship either.
Re: The Case for Managed Domains
Posted by Luca Toscano <to...@gmail.com>.
I am a bit ignorant about mod_bmx so I'd need to ask some follow up
questions: how is it going to solve Stefan's points? As a far as I can see
it is a very nice architecture to query metrics and process them via
plugins, so I'd be super glad to see it integrated to power up our status
page but I don't see how it could help in abstracting properties across
vhosts for example. Can anybody give me more info?
Thanks!
Luca
2018-02-04 15:57 GMT+01:00 Jim Jagielski <ji...@jagunet.com>:
> +1
>
>
> On Feb 4, 2018, at 7:35 AM, Steffen <in...@apachelounge.com> wrote:
>
> There is already Basic Management eXtensions for Apache : mod_bmx is
> handling vhosts.
>
> https://github.com/hyperic/mod_bmx/
>
> Using it on windows with mrtg and mod_watch to create nice graphs per
> vhost.
>
> An issue is that it is not collecting incoming traffic, Bill(William Rowe)
> is looking for a fix.
>
> I am +1 to adopt mod_bmx and use it as a framework and maybe some of
> Stefan proposals we can fit in.
>
> When I recall Bill did already propose to adopt some time ago.
>
>
> Op 4 feb. 2018 om 12:51 heeft Stefan Eissing <st...@greenbytes.de>
> het volgende geschreven:
>
> (Apart from my direct comments in my previous reply in the ServerUID
> discussion,
> I offer a little essay about my motivation in this topic for the
> interested, as it
> is related to the mod_md design I did. It is long. Please feel free to
> ignore it.)
>
>
> When thinking about adding ACME support to Apache, I felt that the server
> lacked
> the concept of something "above" VirtualHosts. Something that keeps common
> properties that some VirtualHosts have in common.
>
>
> The Occasional Admin
> --------------------
> My example was the Apache instance I myself had kept on running over the
> years,
> that has a very limited set of DNS domains it manages. What I had always
> disliked
> is when I ended up copy&pasting configuration directives between different
> VirtualHosts.
>
> I say "ended up", because there are other options like macros or includes,
> but
> for an occasional site admin like myself, those do not work that well. If
> I go
> in to fix something, it will work best with some local changes to the
> VirtualHost
> that I'd like to change/fix. Editing includes/macros is dangerous as I
> have no
> good feeling if this will break other VirtualHosts that also include/use
> this,
> but I have forgotten about that dependency. I only edit the server every
> few
> months or so.
>
> Then I use Ubuntu, which already comes with several includes. But editing
> those is worse, since I am never certain if edits get replaced on update or
> if updates get ignored due to my edits. I could find this all out, but the
> time I am willing to spend on this is limited.
>
> I consider myself a very average httpd admin.
>
>
> The Arrival of SSL
> ------------------
> This all started when SSL was not really a topic for small sites like mine.
> There was 1 VirtualHost for every DNS name served (plus some aliases).
> Changes
> almost always affected only once vhost at a time.
>
> When https: became relevant (and certificates affordable) things got more
> complicated. It doubled the number of VirtualHosts as most of the sites
> needed
> be be available via http: and https: for the same resources (some
> resources only
> to be available on https:).
>
> Since it took some effort to get a certificate, I got few certs with
> several
> alt-names. This meant the same SSL configurations copy&paste between
> vhosts.
> Again, yes, Includes to the rescue. Did some of them, never was happy.
>
> Now, a VirtualHost in my server no longer was the DNS domain. Instead a DNS
> domain was made of several vhosts and, if I did it correctly, the common
> parts were shared and the other parts stayed separated. It worked, but it
> did not feel very natural.
>
> See: as the admin, I do not look at resources for VirtualHosts. I *have*
> different sets of resources and need them to be served in the right way.
> A vhost is just a vessel to do that. And the vessel no longer could contain
> a complete set of resources. I needed to split it over several vessels.
>
>
> Managed Domains
> ---------------
> This was the motivation to design something that aggregates the common
> things
> for several VirtualHosts into something new. The name I came up with for
> that is a "Managed Domain".
>
> "Domain" as it is mainly identified by a DNS name. "Managed" because the
> server should offer some nice defaults on how to serve it and take care
> about things.
> aspects like
>
> Obviously it was made to care about the domain's certificate. Another
> thing is that you can specify that resources in this domain require being
> served via https: only.
>
> There are other things that would fit well into this concept. For example,
> settings and locations for access logging would be nice to have. IO Stats
> as well.
>
> Locations would be really useful.
>
>
> Alternative
> -----------
> Instead of enhancing Managed Domains, one could also go about extending
> VirtualHosts to allow better management of common resource sets. My patch
> to allow address lists in SSLEngine goes in that direction. The drawback
> is that when one overloads existing directives with "special sauce", it
> has limits and can be easily confusing.
>
>
> Conclusion
> ----------
> I hope to have made the case for a concept "above" vhosts. If that is
> an extension of what mod_md currently offers or needs to be reshaped
> is not that important. I merely argue that we need such a thing and
> current config concepts and helpers are not enough.
>
>
> Now, go and enjoy your definitely-not-soccer event...
>
> Cheers,
>
> Stefan
>
>
>
>
Re: The Case for Managed Domains
Posted by Jim Jagielski <ji...@jaguNET.com>.
+1
> On Feb 4, 2018, at 7:35 AM, Steffen <in...@apachelounge.com> wrote:
>
> There is already Basic Management eXtensions for Apache : mod_bmx is handling vhosts.
>
> https://github.com/hyperic/mod_bmx/ <https://github.com/hyperic/mod_bmx/>
>
> Using it on windows with mrtg and mod_watch to create nice graphs per vhost.
>
> An issue is that it is not collecting incoming traffic, Bill(William Rowe) is looking for a fix.
>
> I am +1 to adopt mod_bmx and use it as a framework and maybe some of Stefan proposals we can fit in.
>
> When I recall Bill did already propose to adopt some time ago.
>
>
> Op 4 feb. 2018 om 12:51 heeft Stefan Eissing <stefan.eissing@greenbytes.de <ma...@greenbytes.de>> het volgende geschreven:
>
>> (Apart from my direct comments in my previous reply in the ServerUID discussion,
>> I offer a little essay about my motivation in this topic for the interested, as it
>> is related to the mod_md design I did. It is long. Please feel free to ignore it.)
>>
>>
>> When thinking about adding ACME support to Apache, I felt that the server lacked
>> the concept of something "above" VirtualHosts. Something that keeps common
>> properties that some VirtualHosts have in common.
>>
>>
>> The Occasional Admin
>> --------------------
>> My example was the Apache instance I myself had kept on running over the years,
>> that has a very limited set of DNS domains it manages. What I had always disliked
>> is when I ended up copy&pasting configuration directives between different
>> VirtualHosts.
>>
>> I say "ended up", because there are other options like macros or includes, but
>> for an occasional site admin like myself, those do not work that well. If I go
>> in to fix something, it will work best with some local changes to the VirtualHost
>> that I'd like to change/fix. Editing includes/macros is dangerous as I have no
>> good feeling if this will break other VirtualHosts that also include/use this,
>> but I have forgotten about that dependency. I only edit the server every few
>> months or so.
>>
>> Then I use Ubuntu, which already comes with several includes. But editing
>> those is worse, since I am never certain if edits get replaced on update or
>> if updates get ignored due to my edits. I could find this all out, but the
>> time I am willing to spend on this is limited.
>>
>> I consider myself a very average httpd admin.
>>
>>
>> The Arrival of SSL
>> ------------------
>> This all started when SSL was not really a topic for small sites like mine.
>> There was 1 VirtualHost for every DNS name served (plus some aliases). Changes
>> almost always affected only once vhost at a time.
>>
>> When https: became relevant (and certificates affordable) things got more
>> complicated. It doubled the number of VirtualHosts as most of the sites needed
>> be be available via http: and https: for the same resources (some resources only
>> to be available on https:).
>>
>> Since it took some effort to get a certificate, I got few certs with several
>> alt-names. This meant the same SSL configurations copy&paste between vhosts.
>> Again, yes, Includes to the rescue. Did some of them, never was happy.
>>
>> Now, a VirtualHost in my server no longer was the DNS domain. Instead a DNS
>> domain was made of several vhosts and, if I did it correctly, the common
>> parts were shared and the other parts stayed separated. It worked, but it
>> did not feel very natural.
>>
>> See: as the admin, I do not look at resources for VirtualHosts. I *have*
>> different sets of resources and need them to be served in the right way.
>> A vhost is just a vessel to do that. And the vessel no longer could contain
>> a complete set of resources. I needed to split it over several vessels.
>>
>>
>> Managed Domains
>> ---------------
>> This was the motivation to design something that aggregates the common things
>> for several VirtualHosts into something new. The name I came up with for
>> that is a "Managed Domain".
>>
>> "Domain" as it is mainly identified by a DNS name. "Managed" because the
>> server should offer some nice defaults on how to serve it and take care
>> about things.
>> aspects like
>>
>> Obviously it was made to care about the domain's certificate. Another
>> thing is that you can specify that resources in this domain require being
>> served via https: only.
>>
>> There are other things that would fit well into this concept. For example,
>> settings and locations for access logging would be nice to have. IO Stats
>> as well.
>>
>> Locations would be really useful.
>>
>>
>> Alternative
>> -----------
>> Instead of enhancing Managed Domains, one could also go about extending
>> VirtualHosts to allow better management of common resource sets. My patch
>> to allow address lists in SSLEngine goes in that direction. The drawback
>> is that when one overloads existing directives with "special sauce", it
>> has limits and can be easily confusing.
>>
>>
>> Conclusion
>> ----------
>> I hope to have made the case for a concept "above" vhosts. If that is
>> an extension of what mod_md currently offers or needs to be reshaped
>> is not that important. I merely argue that we need such a thing and
>> current config concepts and helpers are not enough.
>>
>>
>> Now, go and enjoy your definitely-not-soccer event...
>>
>> Cheers,
>>
>> Stefan
>>
>>
Re: The Case for Managed Domains
Posted by Steffen <in...@apachelounge.com>.
There is already Basic Management eXtensions for Apache : mod_bmx is handling vhosts.
https://github.com/hyperic/mod_bmx/
Using it on windows with mrtg and mod_watch to create nice graphs per vhost.
An issue is that it is not collecting incoming traffic, Bill(William Rowe) is looking for a fix.
I am +1 to adopt mod_bmx and use it as a framework and maybe some of Stefan proposals we can fit in.
When I recall Bill did already propose to adopt some time ago.
> Op 4 feb. 2018 om 12:51 heeft Stefan Eissing <st...@greenbytes.de> het volgende geschreven:
>
> (Apart from my direct comments in my previous reply in the ServerUID discussion,
> I offer a little essay about my motivation in this topic for the interested, as it
> is related to the mod_md design I did. It is long. Please feel free to ignore it.)
>
>
> When thinking about adding ACME support to Apache, I felt that the server lacked
> the concept of something "above" VirtualHosts. Something that keeps common
> properties that some VirtualHosts have in common.
>
>
> The Occasional Admin
> --------------------
> My example was the Apache instance I myself had kept on running over the years,
> that has a very limited set of DNS domains it manages. What I had always disliked
> is when I ended up copy&pasting configuration directives between different
> VirtualHosts.
>
> I say "ended up", because there are other options like macros or includes, but
> for an occasional site admin like myself, those do not work that well. If I go
> in to fix something, it will work best with some local changes to the VirtualHost
> that I'd like to change/fix. Editing includes/macros is dangerous as I have no
> good feeling if this will break other VirtualHosts that also include/use this,
> but I have forgotten about that dependency. I only edit the server every few
> months or so.
>
> Then I use Ubuntu, which already comes with several includes. But editing
> those is worse, since I am never certain if edits get replaced on update or
> if updates get ignored due to my edits. I could find this all out, but the
> time I am willing to spend on this is limited.
>
> I consider myself a very average httpd admin.
>
>
> The Arrival of SSL
> ------------------
> This all started when SSL was not really a topic for small sites like mine.
> There was 1 VirtualHost for every DNS name served (plus some aliases). Changes
> almost always affected only once vhost at a time.
>
> When https: became relevant (and certificates affordable) things got more
> complicated. It doubled the number of VirtualHosts as most of the sites needed
> be be available via http: and https: for the same resources (some resources only
> to be available on https:).
>
> Since it took some effort to get a certificate, I got few certs with several
> alt-names. This meant the same SSL configurations copy&paste between vhosts.
> Again, yes, Includes to the rescue. Did some of them, never was happy.
>
> Now, a VirtualHost in my server no longer was the DNS domain. Instead a DNS
> domain was made of several vhosts and, if I did it correctly, the common
> parts were shared and the other parts stayed separated. It worked, but it
> did not feel very natural.
>
> See: as the admin, I do not look at resources for VirtualHosts. I *have*
> different sets of resources and need them to be served in the right way.
> A vhost is just a vessel to do that. And the vessel no longer could contain
> a complete set of resources. I needed to split it over several vessels.
>
>
> Managed Domains
> ---------------
> This was the motivation to design something that aggregates the common things
> for several VirtualHosts into something new. The name I came up with for
> that is a "Managed Domain".
>
> "Domain" as it is mainly identified by a DNS name. "Managed" because the
> server should offer some nice defaults on how to serve it and take care
> about things.
> aspects like
>
> Obviously it was made to care about the domain's certificate. Another
> thing is that you can specify that resources in this domain require being
> served via https: only.
>
> There are other things that would fit well into this concept. For example,
> settings and locations for access logging would be nice to have. IO Stats
> as well.
>
> Locations would be really useful.
>
>
> Alternative
> -----------
> Instead of enhancing Managed Domains, one could also go about extending
> VirtualHosts to allow better management of common resource sets. My patch
> to allow address lists in SSLEngine goes in that direction. The drawback
> is that when one overloads existing directives with "special sauce", it
> has limits and can be easily confusing.
>
>
> Conclusion
> ----------
> I hope to have made the case for a concept "above" vhosts. If that is
> an extension of what mod_md currently offers or needs to be reshaped
> is not that important. I merely argue that we need such a thing and
> current config concepts and helpers are not enough.
>
>
> Now, go and enjoy your definitely-not-soccer event...
>
> Cheers,
>
> Stefan
>
>
Re: The Case for Managed Domains
Posted by Eric Covener <co...@gmail.com>.
On Fri, Feb 9, 2018 at 7:25 AM, Yann Ylavic <yl...@gmail.com> wrote:
> On Fri, Feb 9, 2018 at 12:04 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>>
>>> Am 06.02.2018 um 08:13 schrieb William A Rowe Jr <wr...@rowe-clan.net>:
>>>
>>> Forgive the top-post, Gmail app sucks.
>>>
>>> Thanks for taking the time for tl;dr - it sums up the current situation really well.
>>>
>>> I pointed out some months ago that the matching foo in vhosts is weak, since we have a 1:1 ip:port relationship. We determined that can change in the next iteration to compare a list we already keep. It seems also evident that a vhost could have more than one ServerName as long as we track the canonical name that the request presented.
>>>
>>> Considering your cases, I'm wondering if the simplest thing is to promote <VirtualHost > to a preprocessed, but fully nestable context on trunk?
>>
>> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>>
>> <VirtualHost *:NNN>
>> ServerName example.org
>> <VirtualHost *:80>
>> </VirtualHost>
>> <VirtualHost *:443>
>> </VirtualHost>
>> </VirtualHost>
>>
>> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
>
> The <VirualHost> directive could be relaxed to also take no arg.
> Since at least one is required by now, it wouldn't break anything.
>
>>
>> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>>
>> <ManagedDomain example.org>
>> <VirtualHost *:80>
>> </VirtualHost>
>> <VirtualHost *:443>
>> </VirtualHost>
>> </ManagedDomain>
>>
>> Implementation-wise:
>> * an MD is basically a server_rec plus additional props
>> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
>> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
>> * leaf server_rec->child is NULL
>> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
>>
>> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
>> - Modules should see no change either
>> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>>
>> What do you think?
>
> I like these semantics.
>
> But I'd prefer <VirtualHost> as root, <ManagedDomain ...> is too
> "domain" centric IMHO (all the inner vhosts are necessarily bound to a
> domain).
> With <VirtualHost "some.domain"> or <VirtualHost> + ServerName +
> WhatEverMakesSenseForAllInners we could achieve the same and more
> (including a common <ManagedDomain> why not), while individual inner
> vhosts can have their own domain (if grouping is based on something
> else).
> So I find root <VirtualHost> more generic as a container, we could
> want another name too (Service, Container, ...).
There is infrequently used special handling today for e.g.
<virtualhost example.com> that trumps some of the typical NVH handling
if memory serves.
Maybe that could be shored up or even preferred if it helps. That
still leaves having inner directives that might want to be
port-specific, I just don't know how common enough that is to not
handle it one at a time with logic (SSL) or expressions/if.
I have colleagues that work on a CERN-based legacy server that allows
many directives to take a Host header argument, but it has no config
sections at all.
Perhaps we could provide a simplified <if> that took list of hosts
instead of nesting VH'es or similar sections? But it still has the
problem of per-dir directives only.
--
Eric Covener
covener@gmail.com
Re: The Case for Managed Domains
Posted by Stefan Eissing <st...@greenbytes.de>.
Am 09.02.2018 um 21:21 schrieb Roy T. Fielding <fi...@gbiv.com>:
>> On Feb 9, 2018, at 12:06 PM, Stefan Eissing <st...@greenbytes.de> wrote:
>>> Am 09.02.2018 um 16:12 schrieb Daniel <df...@gmail.com>:
>>>
>>> I'm getting lost.
>>>
>>> What would VirtualServer tag mean exactly?
>>>
>>> Thanks in advance and apologies for my slowness :)
>>
>> The intention of VirtualServer is to solve common properties
>> for a range of VirtualHosts. It's a level between the global
>> settings and individual vhosts. Think of it as having several
>> sets of (nowadays) global configurations for a number of vhosts.
>
> Hmm, I think reuse of that VS name is confusing. How about
> HostSet, HostGroup, GroupHost, or ServerGroup?
You may know by now that I truly enjoy naming discussions here...;-) All those names are fine with me (except GroupHost).
> FWIW, I think it is terribly poor practice to serve the same resources
> via both https and http. Setting an auto-redirect would make more sense.
Agreed, in general. There is the case of a server migration where temporarily both serve the same. Or, you serve a tls terminator on another port.
-Stefan
>
> ....Roy
>
Re: The Case for Managed Domains
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
> On Feb 9, 2018, at 12:06 PM, Stefan Eissing <st...@greenbytes.de> wrote:
>> Am 09.02.2018 um 16:12 schrieb Daniel <df...@gmail.com>:
>>
>> I'm getting lost.
>>
>> What would VirtualServer tag mean exactly?
>>
>> Thanks in advance and apologies for my slowness :)
>
> The intention of VirtualServer is to solve common properties
> for a range of VirtualHosts. It's a level between the global
> settings and individual vhosts. Think of it as having several
> sets of (nowadays) global configurations for a number of vhosts.
Hmm, I think reuse of that VS name is confusing. How about
HostSet, HostGroup, GroupHost, or ServerGroup?
FWIW, I think it is terribly poor practice to serve the same resources
via both https and http. Setting an auto-redirect would make more sense.
....Roy
Re: The Case for Managed Domains
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.02.2018 um 16:12 schrieb Daniel <df...@gmail.com>:
>
> I'm getting lost.
>
> What would VirtualServer tag mean exactly?
>
> Thanks in advance and apologies for my slowness :)
The intention of VirtualServer is to solve common properties
for a range of VirtualHosts. It's a level between the global
settings and individual vhosts. Think of it as having several
sets of (nowadays) global configurations for a number of vhosts.
Examples:
1. Serving the same resources over http: and https:
<VirtualServer example1>
ServerName example.org
ServerAlias www.example.org
DocumentRoot /srv/www/virtual/example.org/htdocs
<Location /bla>
...
</Location>
<VirtualHost *.80>
</VirtualHost>
<VirtualHost *.443>
SSLEngine on
...
</VirtualHost>
</VirtualServer>
2. Having common settings for a customer
<VirtualServer customer2>
ServerAdmin admin@customer2.com
...
<VirtualHost *:443>
ServerName www1.customer2.com
...
</VirtualHost>
<VirtualHost *:443>
ServerName www2.customer2.com
...
</VirtualHost>
</VirtualServer>
This all could instead be simulated with clever use of Include or Macros.
But the inheritance by a VirtualServer is better to read and understand,
and so, I argue, easier to get "right" and maintain.
Also, we could add directives that enforce limits on what the vhosts
can do. Imagine a hoster giving a customers the possibilities to define
their vhosts themselves. They all get included in a VirtualServer that
limits where DocumentRoots can point to, which directives are available
etc.
Hope this explains the gist of this.
-Stefan
> 2018-02-09 13:42 GMT+01:00 Stefan Eissing <st...@greenbytes.de>:
>>
>>
>>> Am 09.02.2018 um 13:25 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>> On Fri, Feb 9, 2018 at 12:04 PM, Stefan Eissing
>>> <st...@greenbytes.de> wrote:
>>>>
>>>>
>>>>> Am 06.02.2018 um 08:13 schrieb William A Rowe Jr <wr...@rowe-clan.net>:
>>>>>
>>>>> Forgive the top-post, Gmail app sucks.
>>>>>
>>>>> Thanks for taking the time for tl;dr - it sums up the current situation really well.
>>>>>
>>>>> I pointed out some months ago that the matching foo in vhosts is weak, since we have a 1:1 ip:port relationship. We determined that can change in the next iteration to compare a list we already keep. It seems also evident that a vhost could have more than one ServerName as long as we track the canonical name that the request presented.
>>>>>
>>>>> Considering your cases, I'm wondering if the simplest thing is to promote <VirtualHost > to a preprocessed, but fully nestable context on trunk?
>>>>
>>>> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>>>>
>>>> <VirtualHost *:NNN>
>>>> ServerName example.org
>>>> <VirtualHost *:80>
>>>> </VirtualHost>
>>>> <VirtualHost *:443>
>>>> </VirtualHost>
>>>> </VirtualHost>
>>>>
>>>> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
>>>
>>> The <VirualHost> directive could be relaxed to also take no arg.
>>> Since at least one is required by now, it wouldn't break anything.
>>>
>>>>
>>>> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>>>>
>>>> <ManagedDomain example.org>
>>>> <VirtualHost *:80>
>>>> </VirtualHost>
>>>> <VirtualHost *:443>
>>>> </VirtualHost>
>>>> </ManagedDomain>
>>>>
>>>> Implementation-wise:
>>>> * an MD is basically a server_rec plus additional props
>>>> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
>>>> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
>>>> * leaf server_rec->child is NULL
>>>> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
>>>>
>>>> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
>>>> - Modules should see no change either
>>>> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>>>>
>>>> What do you think?
>>>
>>> I like these semantics.
>>>
>>> But I'd prefer <VirtualHost> as root, <ManagedDomain ...> is too
>>> "domain" centric IMHO (all the inner vhosts are necessarily bound to a
>>> domain).
>>> With <VirtualHost "some.domain"> or <VirtualHost> + ServerName +
>>> WhatEverMakesSenseForAllInners we could achieve the same and more
>>> (including a common <ManagedDomain> why not), while individual inner
>>> vhosts can have their own domain (if grouping is based on something
>>> else).
>>> So I find root <VirtualHost> more generic as a container, we could
>>> want another name too (Service, Container, ...).
>>
>> Now, there is a nice naming debate in the makings... ;-)
>>
>> I appreciate that you like the concept. I agree that groupings that do
>> not have a common DNS make sense too and should be possible (first thing
>> comes to mind is a grouping by customers).
>>
>> Since we are in "VirtualHost" land anyway, I can agree to
>>
>> <VirtualServer "name">
>>
>> (I like admin supplied unique names - they are useful.:)
>>
>> -Stefan
>>
>>
>
>
>
> --
> Daniel Ferradal
> IT Specialist
>
> email dferradal at gmail.com
> linkedin es.linkedin.com/in/danielferradal
Re: The Case for Managed Domains
Posted by Daniel <df...@gmail.com>.
I'm getting lost.
What would VirtualServer tag mean exactly?
Thanks in advance and apologies for my slowness :)
2018-02-09 13:42 GMT+01:00 Stefan Eissing <st...@greenbytes.de>:
>
>
>> Am 09.02.2018 um 13:25 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> On Fri, Feb 9, 2018 at 12:04 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>>
>>>
>>>> Am 06.02.2018 um 08:13 schrieb William A Rowe Jr <wr...@rowe-clan.net>:
>>>>
>>>> Forgive the top-post, Gmail app sucks.
>>>>
>>>> Thanks for taking the time for tl;dr - it sums up the current situation really well.
>>>>
>>>> I pointed out some months ago that the matching foo in vhosts is weak, since we have a 1:1 ip:port relationship. We determined that can change in the next iteration to compare a list we already keep. It seems also evident that a vhost could have more than one ServerName as long as we track the canonical name that the request presented.
>>>>
>>>> Considering your cases, I'm wondering if the simplest thing is to promote <VirtualHost > to a preprocessed, but fully nestable context on trunk?
>>>
>>> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>>>
>>> <VirtualHost *:NNN>
>>> ServerName example.org
>>> <VirtualHost *:80>
>>> </VirtualHost>
>>> <VirtualHost *:443>
>>> </VirtualHost>
>>> </VirtualHost>
>>>
>>> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
>>
>> The <VirualHost> directive could be relaxed to also take no arg.
>> Since at least one is required by now, it wouldn't break anything.
>>
>>>
>>> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>>>
>>> <ManagedDomain example.org>
>>> <VirtualHost *:80>
>>> </VirtualHost>
>>> <VirtualHost *:443>
>>> </VirtualHost>
>>> </ManagedDomain>
>>>
>>> Implementation-wise:
>>> * an MD is basically a server_rec plus additional props
>>> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
>>> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
>>> * leaf server_rec->child is NULL
>>> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
>>>
>>> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
>>> - Modules should see no change either
>>> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>>>
>>> What do you think?
>>
>> I like these semantics.
>>
>> But I'd prefer <VirtualHost> as root, <ManagedDomain ...> is too
>> "domain" centric IMHO (all the inner vhosts are necessarily bound to a
>> domain).
>> With <VirtualHost "some.domain"> or <VirtualHost> + ServerName +
>> WhatEverMakesSenseForAllInners we could achieve the same and more
>> (including a common <ManagedDomain> why not), while individual inner
>> vhosts can have their own domain (if grouping is based on something
>> else).
>> So I find root <VirtualHost> more generic as a container, we could
>> want another name too (Service, Container, ...).
>
> Now, there is a nice naming debate in the makings... ;-)
>
> I appreciate that you like the concept. I agree that groupings that do
> not have a common DNS make sense too and should be possible (first thing
> comes to mind is a grouping by customers).
>
> Since we are in "VirtualHost" land anyway, I can agree to
>
> <VirtualServer "name">
>
> (I like admin supplied unique names - they are useful.:)
>
> -Stefan
>
>
--
Daniel Ferradal
IT Specialist
email dferradal at gmail.com
linkedin es.linkedin.com/in/danielferradal
Re: The Case for Managed Domains
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.02.2018 um 13:25 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Feb 9, 2018 at 12:04 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>>
>>> Am 06.02.2018 um 08:13 schrieb William A Rowe Jr <wr...@rowe-clan.net>:
>>>
>>> Forgive the top-post, Gmail app sucks.
>>>
>>> Thanks for taking the time for tl;dr - it sums up the current situation really well.
>>>
>>> I pointed out some months ago that the matching foo in vhosts is weak, since we have a 1:1 ip:port relationship. We determined that can change in the next iteration to compare a list we already keep. It seems also evident that a vhost could have more than one ServerName as long as we track the canonical name that the request presented.
>>>
>>> Considering your cases, I'm wondering if the simplest thing is to promote <VirtualHost > to a preprocessed, but fully nestable context on trunk?
>>
>> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>>
>> <VirtualHost *:NNN>
>> ServerName example.org
>> <VirtualHost *:80>
>> </VirtualHost>
>> <VirtualHost *:443>
>> </VirtualHost>
>> </VirtualHost>
>>
>> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
>
> The <VirualHost> directive could be relaxed to also take no arg.
> Since at least one is required by now, it wouldn't break anything.
>
>>
>> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>>
>> <ManagedDomain example.org>
>> <VirtualHost *:80>
>> </VirtualHost>
>> <VirtualHost *:443>
>> </VirtualHost>
>> </ManagedDomain>
>>
>> Implementation-wise:
>> * an MD is basically a server_rec plus additional props
>> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
>> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
>> * leaf server_rec->child is NULL
>> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
>>
>> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
>> - Modules should see no change either
>> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>>
>> What do you think?
>
> I like these semantics.
>
> But I'd prefer <VirtualHost> as root, <ManagedDomain ...> is too
> "domain" centric IMHO (all the inner vhosts are necessarily bound to a
> domain).
> With <VirtualHost "some.domain"> or <VirtualHost> + ServerName +
> WhatEverMakesSenseForAllInners we could achieve the same and more
> (including a common <ManagedDomain> why not), while individual inner
> vhosts can have their own domain (if grouping is based on something
> else).
> So I find root <VirtualHost> more generic as a container, we could
> want another name too (Service, Container, ...).
Now, there is a nice naming debate in the makings... ;-)
I appreciate that you like the concept. I agree that groupings that do
not have a common DNS make sense too and should be possible (first thing
comes to mind is a grouping by customers).
Since we are in "VirtualHost" land anyway, I can agree to
<VirtualServer "name">
(I like admin supplied unique names - they are useful.:)
-Stefan
Re: The Case for Managed Domains
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Feb 9, 2018 at 12:04 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
>
>
>> Am 06.02.2018 um 08:13 schrieb William A Rowe Jr <wr...@rowe-clan.net>:
>>
>> Forgive the top-post, Gmail app sucks.
>>
>> Thanks for taking the time for tl;dr - it sums up the current situation really well.
>>
>> I pointed out some months ago that the matching foo in vhosts is weak, since we have a 1:1 ip:port relationship. We determined that can change in the next iteration to compare a list we already keep. It seems also evident that a vhost could have more than one ServerName as long as we track the canonical name that the request presented.
>>
>> Considering your cases, I'm wondering if the simplest thing is to promote <VirtualHost > to a preprocessed, but fully nestable context on trunk?
>
> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>
> <VirtualHost *:NNN>
> ServerName example.org
> <VirtualHost *:80>
> </VirtualHost>
> <VirtualHost *:443>
> </VirtualHost>
> </VirtualHost>
>
> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
The <VirualHost> directive could be relaxed to also take no arg.
Since at least one is required by now, it wouldn't break anything.
>
> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>
> <ManagedDomain example.org>
> <VirtualHost *:80>
> </VirtualHost>
> <VirtualHost *:443>
> </VirtualHost>
> </ManagedDomain>
>
> Implementation-wise:
> * an MD is basically a server_rec plus additional props
> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
> * leaf server_rec->child is NULL
> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
>
> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
> - Modules should see no change either
> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>
> What do you think?
I like these semantics.
But I'd prefer <VirtualHost> as root, <ManagedDomain ...> is too
"domain" centric IMHO (all the inner vhosts are necessarily bound to a
domain).
With <VirtualHost "some.domain"> or <VirtualHost> + ServerName +
WhatEverMakesSenseForAllInners we could achieve the same and more
(including a common <ManagedDomain> why not), while individual inner
vhosts can have their own domain (if grouping is based on something
else).
So I find root <VirtualHost> more generic as a container, we could
want another name too (Service, Container, ...).
Re: The Case for Managed Domains
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.02.2018 um 13:28 schrieb Eric Covener <co...@gmail.com>:
>
>> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>>
>> <VirtualHost *:NNN>
>> ServerName example.org
>> <VirtualHost *:80>
>> </VirtualHost>
>> <VirtualHost *:443>
>> </VirtualHost>
>> </VirtualHost>
>>
>> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
>
> I would prefer we just make <virtualhost *:*> or <virtualhost *:80
> *:443> work a little better.
> But in the hypothetical nested example, the address could be omitted,
> use *:*, or use *:80 *:443. I don't like the churn for mods with
> nesting though, since they have server config merge callbacks that
> might be subtly wrong.
>
>>
>> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>>
>> <ManagedDomain example.org>
>> <VirtualHost *:80>
>> </VirtualHost>
>> <VirtualHost *:443>
>> </VirtualHost>
>> </ManagedDomain>
>>
>> Implementation-wise:
>> * an MD is basically a server_rec plus additional props
>> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
>> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
>> * leaf server_rec->child is NULL
>> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
>
> Are directives valid in VH implicitly valid in MD and using server
> config merging? Or are normal directives not anticipated in that
> scope?
> It seems like more for users and module authors to worry about.
>
>>
>> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
>> - Modules should see no change either
>> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>>
>> What do you think?
>
> I would personally rather see no additional nesting and a replacement
> or extension for <virtualhost> syntax and mapping, along with a way
> forward (by convention) to make directives behave a little better in
> <VH *:80 *:443>, but most of my users/users are a very low # of VH'es
> so it's not really something I am focused on / tuned into.
My experiment in trunk with a
<VirtualHost *:80 *:443>
SSLEngine *:443
</VirtualHost>
worked. But SSLEngine might not be the only directive where you
need some extra "magic" to make it apply to the desired vhost variation.
For clarity, I am now more in favour of nesting vhosts/domains/whatever,
since this makes it more clear when which config applies. Our server
config merging between base and vhosts is well understood and this
would just extend the mechanism.
-Stefan
Re: The Case for Managed Domains
Posted by Eric Covener <co...@gmail.com>.
> That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
>
> <VirtualHost *:NNN>
> ServerName example.org
> <VirtualHost *:80>
> </VirtualHost>
> <VirtualHost *:443>
> </VirtualHost>
> </VirtualHost>
>
> What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
I would prefer we just make <virtualhost *:*> or <virtualhost *:80
*:443> work a little better.
But in the hypothetical nested example, the address could be omitted,
use *:*, or use *:80 *:443. I don't like the churn for mods with
nesting though, since they have server config merge callbacks that
might be subtly wrong.
>
> From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
>
> <ManagedDomain example.org>
> <VirtualHost *:80>
> </VirtualHost>
> <VirtualHost *:443>
> </VirtualHost>
> </ManagedDomain>
>
> Implementation-wise:
> * an MD is basically a server_rec plus additional props
> * server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
> * server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
> * leaf server_rec->child is NULL
> * no nesting of MDs foreseen, disabled, no nesting of VirtualHost
Are directives valid in VH implicitly valid in MD and using server
config merging? Or are normal directives not anticipated in that
scope?
It seems like more for users and module authors to worry about.
>
> - VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
> - Modules should see no change either
> - The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
>
> What do you think?
I would personally rather see no additional nesting and a replacement
or extension for <virtualhost> syntax and mapping, along with a way
forward (by convention) to make directives behave a little better in
<VH *:80 *:443>, but most of my users/users are a very low # of VH'es
so it's not really something I am focused on / tuned into.
Re: The Case for Managed Domains
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 06.02.2018 um 08:13 schrieb William A Rowe Jr <wr...@rowe-clan.net>:
>
> Forgive the top-post, Gmail app sucks.
>
> Thanks for taking the time for tl;dr - it sums up the current situation really well.
>
> I pointed out some months ago that the matching foo in vhosts is weak, since we have a 1:1 ip:port relationship. We determined that can change in the next iteration to compare a list we already keep. It seems also evident that a vhost could have more than one ServerName as long as we track the canonical name that the request presented.
>
> Considering your cases, I'm wondering if the simplest thing is to promote <VirtualHost > to a preprocessed, but fully nestable context on trunk?
That is one possible way to address this. However, since VirtualHost requires at least one address, this would be confusing:
<VirtualHost *:NNN>
ServerName example.org
<VirtualHost *:80>
</VirtualHost>
<VirtualHost *:443>
</VirtualHost>
</VirtualHost>
What would NNN be or what would it mean? Would it also be inherited by the nested vhosts? This is confusing.
From usability point of view, it is superior to introduce a new "section" that can keep common locations and other settings for sharing. The implementation is more painful, though. Unsurprisingly, I think of something like this:
<ManagedDomain example.org>
<VirtualHost *:80>
</VirtualHost>
<VirtualHost *:443>
</VirtualHost>
</ManagedDomain>
Implementation-wise:
* an MD is basically a server_rec plus additional props
* server_rec gets 2 new members "server_rec *child, *sibling" to keep contained vhosts
* server_rec->next contains only leaf server_recs -> base_server->next contains all VirtualHosts as before
* leaf server_rec->child is NULL
* no nesting of MDs foreseen, disabled, no nesting of VirtualHost
- VirtualHosts server_rec* can be walked as before and all vhost functions see no change.
- Modules should see no change either
- The MD and VirtualHost hierarchy can be traversed by new code starting from base server.
What do you think?
-Stefan
> One, two, three deep VirtualHost server scopes would provide sharing of config choices between hosts. Because some of these would be Management Scopes, as you suggest, using TAKE_01 would allow an umbrella scope to not actually be attached to an interface at all, but just serve, as the global context does, for inheritance only hosts.
>
> Simplest example would be a well scoped http:// interface with a nested https:// interface, adding some protected ssl-only content mapping and tls config. That would not diminish the outer http:// vhost mapping.
>
> Similar examples would be a wild card SSL cert config with a number of specific content hosts nested within that, sharing the same tls config.
>
> What this doesn't solve is a many:many mapping. I haven't come up with a good answer to that puzzle without resorting to includes and macros, etc.
>
>
>
> On Feb 4, 2018 05:51, "Stefan Eissing" <st...@greenbytes.de> wrote:
>
> When thinking about adding ACME support to Apache, I felt that the server lacked
> the concept of something "above" VirtualHosts. Something that keeps common
> properties that some VirtualHosts have in common.
Re: The Case for Managed Domains
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
Forgive the top-post, Gmail app sucks.
Thanks for taking the time for tl;dr - it sums up the current situation
really well.
I pointed out some months ago that the matching foo in vhosts is weak,
since we have a 1:1 ip:port relationship. We determined that can change in
the next iteration to compare a list we already keep. It seems also evident
that a vhost could have more than one ServerName as long as we track the
canonical name that the request presented.
Considering your cases, I'm wondering if the simplest thing is to promote
<VirtualHost > to a preprocessed, but fully nestable context on trunk?
One, two, three deep VirtualHost server scopes would provide sharing of
config choices between hosts. Because some of these would be Management
Scopes, as you suggest, using TAKE_01 would allow an umbrella scope to not
actually be attached to an interface at all, but just serve, as the global
context does, for inheritance only hosts.
Simplest example would be a well scoped http:// interface with a nested
https:// interface, adding some protected ssl-only content mapping and tls
config. That would not diminish the outer http:// vhost mapping.
Similar examples would be a wild card SSL cert config with a number of
specific content hosts nested within that, sharing the same tls config.
What this doesn't solve is a many:many mapping. I haven't come up with a
good answer to that puzzle without resorting to includes and macros, etc.
On Feb 4, 2018 05:51, "Stefan Eissing" <st...@greenbytes.de> wrote:
When thinking about adding ACME support to Apache, I felt that the server
lacked
the concept of something "above" VirtualHosts. Something that keeps common
properties that some VirtualHosts have in common.
The Case for Managed Domains
Posted by Stefan Eissing <st...@greenbytes.de>.
(Apart from my direct comments in my previous reply in the ServerUID discussion,
I offer a little essay about my motivation in this topic for the interested, as it
is related to the mod_md design I did. It is long. Please feel free to ignore it.)
When thinking about adding ACME support to Apache, I felt that the server lacked
the concept of something "above" VirtualHosts. Something that keeps common
properties that some VirtualHosts have in common.
The Occasional Admin
--------------------
My example was the Apache instance I myself had kept on running over the years,
that has a very limited set of DNS domains it manages. What I had always disliked
is when I ended up copy&pasting configuration directives between different
VirtualHosts.
I say "ended up", because there are other options like macros or includes, but
for an occasional site admin like myself, those do not work that well. If I go
in to fix something, it will work best with some local changes to the VirtualHost
that I'd like to change/fix. Editing includes/macros is dangerous as I have no
good feeling if this will break other VirtualHosts that also include/use this,
but I have forgotten about that dependency. I only edit the server every few
months or so.
Then I use Ubuntu, which already comes with several includes. But editing
those is worse, since I am never certain if edits get replaced on update or
if updates get ignored due to my edits. I could find this all out, but the
time I am willing to spend on this is limited.
I consider myself a very average httpd admin.
The Arrival of SSL
------------------
This all started when SSL was not really a topic for small sites like mine.
There was 1 VirtualHost for every DNS name served (plus some aliases). Changes
almost always affected only once vhost at a time.
When https: became relevant (and certificates affordable) things got more
complicated. It doubled the number of VirtualHosts as most of the sites needed
be be available via http: and https: for the same resources (some resources only
to be available on https:).
Since it took some effort to get a certificate, I got few certs with several
alt-names. This meant the same SSL configurations copy&paste between vhosts.
Again, yes, Includes to the rescue. Did some of them, never was happy.
Now, a VirtualHost in my server no longer was the DNS domain. Instead a DNS
domain was made of several vhosts and, if I did it correctly, the common
parts were shared and the other parts stayed separated. It worked, but it
did not feel very natural.
See: as the admin, I do not look at resources for VirtualHosts. I *have*
different sets of resources and need them to be served in the right way.
A vhost is just a vessel to do that. And the vessel no longer could contain
a complete set of resources. I needed to split it over several vessels.
Managed Domains
---------------
This was the motivation to design something that aggregates the common things
for several VirtualHosts into something new. The name I came up with for
that is a "Managed Domain".
"Domain" as it is mainly identified by a DNS name. "Managed" because the
server should offer some nice defaults on how to serve it and take care
about things.
aspects like
Obviously it was made to care about the domain's certificate. Another
thing is that you can specify that resources in this domain require being
served via https: only.
There are other things that would fit well into this concept. For example,
settings and locations for access logging would be nice to have. IO Stats
as well.
Locations would be really useful.
Alternative
-----------
Instead of enhancing Managed Domains, one could also go about extending
VirtualHosts to allow better management of common resource sets. My patch
to allow address lists in SSLEngine goes in that direction. The drawback
is that when one overloads existing directives with "special sauce", it
has limits and can be easily confusing.
Conclusion
----------
I hope to have made the case for a concept "above" vhosts. If that is
an extension of what mod_md currently offers or needs to be reshaped
is not that important. I merely argue that we need such a thing and
current config concepts and helpers are not enough.
Now, go and enjoy your definitely-not-soccer event...
Cheers,
Stefan
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Sun, Feb 4, 2018 at 3:32 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
>
>> On Feb 2, 2018, at 8:05 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>
>> That's a new directive too, very specialized (though it's your point).
>> Wouldn't it end up being declined to other modules one day or the other?
>> But mainly, rather than ignoring config changes, I think mod_proxy_lb
>> should ignore non-changes.
>> And it already knows how to do that provided the vhost's balancer id
>> does not change unnecessarily...
>
> The main point is that modules are free to determine what they
> consider a unique Vhost; they are free to make whatever assumptions
> are required. Inevitably, it depends on the module and WHY it needs
> to know if a Vhost is "unique" and that will vary. Some will be more
> stringent, others more free. Creating a user land directive which is
> basically for internal module use doesn't help at all.
I agree with that in general, can we agree that mod_proxy_lb wants a
fixed server id on restart and that it'd rather create new SHMs solely
based on its own configuration changes (i.e. lbmethod, # members,
members' loadfactor/status/...)?
This is clearly not what happens currently (besides maybe the number
of members which could make the SHM's size change and cause a reset).
>
> The only issue is that *in this particular case* there is a
> weird use case where the assumption on what makes a
> Vhost unique doesn't match with what they want. One
> option, of course, is to say "Well, you need to figure out
> a way such that when you adjust the config file, the line
> number doesn't change". For such an extremely rare edge
> case, this seems a valid suggestion, rather than directly
> jumping into code and adding fluff and cruft that 99.9999%
> of httpd users will never want and see.
I don't understand why you state it's a weird/edge case.
Start httpd, add a blank line anywhere before the vhost which runs the
balancer, gracefuly restart httpd and see how new SHMs are created for
the new generation, losing all the previous state (load distribution,
errors, stats...).
IMHO the old and new generations should share and preserve the state here.
>
> For example, consider the above. You say "I think mod_proxy_lb
> should ignore non-changes.". What do we do for the admin
> who doesn't feel that way? Or that we want ServerUID to be
> one thing for module foo and another for module bar? Isn't
> it up to the *module* itself to determine what makes sense?
> The module itself is (hopefully!! :) ) designed and architected
> and "logic" with its own specific assumption and requirement
> on what a unique UID needs to be. I would guess that 99.999%
> of the time, any change to that would cause really weird behavior.
If modules want to build this uid for their own use they can already
do that (with whatever they find relevent in the server_rec).
But if they want to reassociate their resources to the same vhost
regardless of changes (irrelevent to them) done on restart, how would
they without the help of the only who knows (the webmaster)? This is
where any automatic thing can't please everyone.
Of course this does not avoid the module to check that the
configuration they care about is still compatible with the resources
they want to reuse, and to do the proper action otherwise, but
anything else is irrevelevent to them.
For instance, for the mod_proxy_lb case, if you add a listening IP or
a ServerAlias I don't see why the balancer should suddenly clear
everything, or refuse to start, should it? Isn't it about backend
connections only?
A quick look at "mod_bmx_vhost.c" (mentioned in the other thread)
shows that it uses "BMXVHostDBMFilename" to identify vhosts, i.e. by
their dbm path. If you want a simple and targeted solution for
mod_proxy_lb, just let the admin configure the SHM file path, that's
the UID. Simple.
Friendly? hmm. Prone to directives proliferation? sure.
>
> This just seems like a heavy-weight "fix" for an extremely
> weird on one-off edge case which is more suitably handled
> by, if an actual code change is really *required*, a small
> targeted patch.
Again, the ServerUID is not a fix from my POV, it's not targeted at
mod_proxy_lb.
And the mod_proxy_lb case is not weird, it's a bug which we should
fix, with or without ServerUID.
But I don't want to push ServerUid against all odds, let's forget
about it and talk about an internal/autmatic "server uid" which
uniquely identifies a vhost and can serve all users (no that weird)
needs.
What would it be for mod_proxy_lb since we are at it? :)
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Feb 5, 2018, at 7:38 AM, Stefan Eissing <st...@greenbytes.de> wrote:
>
>
> 2. Does httpd core expose a VirtualHost Identifier in its API and
> what would the semantic properties of such an identifier be?
>
Yes, it does. It's the server_rec. That contains all the info required
to determine any and all fashion of "unique" Vhost IDs.
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 04.02.2018 um 15:32 schrieb Jim Jagielski <ji...@jaguNET.com>:
>
>
>
>> On Feb 2, 2018, at 8:05 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>
>> That's a new directive too, very specialized (though it's your point).
>> Wouldn't it end up being declined to other modules one day or the other?
>> But mainly, rather than ignoring config changes, I think mod_proxy_lb
>> should ignore non-changes.
>> And it already knows how to do that provided the vhost's balancer id
>> does not change unnecessarily...
>
> The main point is that modules are free to determine what they
> consider a unique Vhost; they are free to make whatever assumptions
> are required. Inevitably, it depends on the module and WHY it needs
> to know if a Vhost is "unique" and that will vary. Some will be more
> stringent, others more free. Creating a user land directive which is
> basically for internal module use doesn't help at all.
No one wants to restrict any module's freedom to choose meaningful
identifier for the data it manages. I think we are all in agreement
to that.
There seem to be two things worth discussion:
1. How can the default behaviour of mod_proxy_lb be changed, so
that site owner with frequent reloads no longer suffer the current
problems.
I think we can get a solution without introducing a new directive.
That would be good for our users.
2. Does httpd core expose a VirtualHost Identifier in its API and
what would the semantic properties of such an identifier be?
The second part is what I and Yann have both argued for. I do not
know if we are in 100% agreement to what it should be, but we both
seem to see use cases where an identifier (for me, an id always is
unique) for a virtual host is useful.
Again, I think that such a server id can be handed out by httpd
without using a directive. That id does not have to satisfy all
uses cases, since modules - as stated above - remain free to use
it or come up with their own.
Cheers,
Stefan
>
> The only issue is that *in this particular case* there is a
> weird use case where the assumption on what makes a
> Vhost unique doesn't match with what they want. One
> option, of course, is to say "Well, you need to figure out
> a way such that when you adjust the config file, the line
> number doesn't change". For such an extremely rare edge
> case, this seems a valid suggestion, rather than directly
> jumping into code and adding fluff and cruft that 99.9999%
> of httpd users will never want and see.
>
> But sometimes, such edge cases are not-so-rare enough that
> we provide code workaround for the normal case, ala the
> singular workaround directive suggested. It's specific, targeted
> and low touch for the vast majority of httpd users and admins.
>
> For example, consider the above. You say "I think mod_proxy_lb
> should ignore non-changes.". What do we do for the admin
> who doesn't feel that way? Or that we want ServerUID to be
> one thing for module foo and another for module bar? Isn't
> it up to the *module* itself to determine what makes sense?
> The module itself is (hopefully!! :) ) designed and architected
> and "logic" with its own specific assumption and requirement
> on what a unique UID needs to be. I would guess that 99.999%
> of the time, any change to that would cause really weird behavior.
>
> This just seems like a heavy-weight "fix" for an extremely
> weird on one-off edge case which is more suitably handled
> by, if an actual code change is really *required*, a small
> targeted patch.
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Feb 2, 2018, at 8:05 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> That's a new directive too, very specialized (though it's your point).
> Wouldn't it end up being declined to other modules one day or the other?
> But mainly, rather than ignoring config changes, I think mod_proxy_lb
> should ignore non-changes.
> And it already knows how to do that provided the vhost's balancer id
> does not change unnecessarily...
The main point is that modules are free to determine what they
consider a unique Vhost; they are free to make whatever assumptions
are required. Inevitably, it depends on the module and WHY it needs
to know if a Vhost is "unique" and that will vary. Some will be more
stringent, others more free. Creating a user land directive which is
basically for internal module use doesn't help at all.
The only issue is that *in this particular case* there is a
weird use case where the assumption on what makes a
Vhost unique doesn't match with what they want. One
option, of course, is to say "Well, you need to figure out
a way such that when you adjust the config file, the line
number doesn't change". For such an extremely rare edge
case, this seems a valid suggestion, rather than directly
jumping into code and adding fluff and cruft that 99.9999%
of httpd users will never want and see.
But sometimes, such edge cases are not-so-rare enough that
we provide code workaround for the normal case, ala the
singular workaround directive suggested. It's specific, targeted
and low touch for the vast majority of httpd users and admins.
For example, consider the above. You say "I think mod_proxy_lb
should ignore non-changes.". What do we do for the admin
who doesn't feel that way? Or that we want ServerUID to be
one thing for module foo and another for module bar? Isn't
it up to the *module* itself to determine what makes sense?
The module itself is (hopefully!! :) ) designed and architected
and "logic" with its own specific assumption and requirement
on what a unique UID needs to be. I would guess that 99.999%
of the time, any change to that would cause really weird behavior.
This just seems like a heavy-weight "fix" for an extremely
weird on one-off edge case which is more suitably handled
by, if an actual code change is really *required*, a small
targeted patch.
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Feb 4, 2018, at 5:49 AM, Stefan Eissing <st...@greenbytes.de> wrote:
>
>
> Initially, ServerUID sounded like a tweak, but now we are talking
> about the identity of a server_rec across reloads and how that
> can be useful.
>
But does it need to be a user Directive? I still say it is not needed
whether or not.
My point is that each module is free to determine what makes
a Vhost "unique", all by itself, across graceful restarts, across
restarts with config file changes, etc... It is all based on
the assumption that, TO THAT MODULE, what "is" a unique
Vhost. And that assumption is not absolute. It depends on
the use case. And modules, via post_config and determining
which post config # it is, can create it's own internal ServerUID
for whatever it needs to use it for. There is no need for this
to be some external ID for the user land.
So even if we create this directive, we will undoubtedly run into
cases where people say "Well, it doesn't do exactly what
I want.... I want ServerUID to change when I do 'foo' or remain
the same when I do 'bar'" and we will add additional fluff
and cruft.
It's a solution looking for a problem.
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 03.02.2018 um 02:05 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Feb 2, 2018 at 5:52 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>> But we already have a unique ID in the form of
>> the actual server struct itself, which contains
>> all the data required to make a vhost "unique"
>
> Sure, my patch or any other way to compute an automatic UID for a
> vhost don't need anything else than the server_rec.
> More exactly the proposed patch needs all the servers because it
> doesn't take the line number into account, and nothing prevents
> multiple exact same vhosts to be configured (totally useless, so an
> implementation detail and not the point here).
>
>>
>> The only "issue" is what makes a host "unique"... IMO,
>> in an environ where the definition line of the vhost changes
>> between restarts is a "new" vhost and that was the assumption
>> that the implementation was based on.
>
> I don't deny it can be a valid assumption for some cases, but I
> observe that it breaks others like "automatic vhost UID should remain
> the same when some totally unrelated line is added before it".
Code exists that wants to persist (and be it shared mem) across
reloads or even restarts for the same (set of) virtual hosts.
Code that wants to re-init on every reload has the generation
and/or timestamps to work with, I assume.
>> *But that assumption can more easily be changed by providing
>> a work-around for that specific use case*
>
> I'm not proposing this directive to address the mod_proxy_lb case, and
> just gave it as an example for an in-tree case where it would be
> useful (at least a new automatic UID is needed I think).
> That case just reminded me that I had a patch somewhere for the
> ServerUID, which is actually part of a larger one for a per-vhost
> scoreboard/logio stuff which I already talked about some times ago
> (with no apparent feedback/interest).
On my ever growing list of needful things is status information about
managed domains. It would be interesting to also show how much traffic
and redirects are being observed on individual server_rec* belonging
to a domain.
> This larger patch really needs a vhost UID for stats to be relevent
> days/month/years later, regardless of minor configuration changes,
> even for things like vhost IP or port (which does not matter too much
> in a local network, not to talk about line numbers...) or
> ServerName/Alias (nothing mandates them for a vhost to work, e.g. IPs
> only and ProxyPreserveHost).
I always think of ServerName as identifier, although the server itself
does not make that assertion. What would be the use of identifying vhosts
*not* based on their ServerName? (Curious)
> Anyway, if you already have an UID in an User Interface to identify a
> vhost, put it in ServerUID for httpd to use it where needed internally
> and/or to "print" that UID anywhere you configured it to (e.g.
> %{SERVER_UID} in access_log, headers, expr, ...).
> At the end of the day, the webmaster really talks about the ServerName
> or this "UID" to refer to a vhost (they can be and even usually are
> the same modulo ports 80/443), but never about the IP or line number
> it once had.
+1
> Sooo :) instead of proposing the change on how we can uniquely
> identify a vhost for mod_proxy_lb only (in an automatic way or not), I
> thought ServerUID could start its upstream life.
+1.
>> instead of creating a
>> whole new Directive for a "generic" server UID which:
>>
>> 1. Implies that the webmaster must know when and how that UID
>> is used internally
I think key is to define a concept of a server identifier that
has clear semantics without knowing the internals.
> How is it different than now, or than anything we could do
> automatically/unconditionaly based on the server_rec?
> This argument sounds to me like we really need the ServerUID, if the
> webmaster really wants httpd to use an ID which does not depend on
> whatever future change to the configuration, (s)he has to provide that
> ID or know the internals.
> The only requirement on the httpd side is that an given ServerUID is
> unique (that should be documented, though the directive name not
> really ambiguous). It shouldn't be too hard for the user, UUID or
> alike.
>
>> 2. Add more Directive cruft to an already bloated collection
>> of directives.
>
> That's a very valid concern! But I think my proposal is quite
> orthogonal to the mod_proxy_lb issue, and it doesn't avoid us to find
> one (or Stefan's multiple) good way to uniquely identify a vhost when
> no ServerUID is specified.
There are directives which introduce "things" in the server, e.g.
"VirtualHost" and there are directives which peek/poke at highly
specific aspects of something, tweaking the defaults. There are
few of the first and a myriad of the latter ones.
Initially, ServerUID sounded like a tweak, but now we are talking
about the identity of a server_rec across reloads and how that
can be useful.
I would like to come to a design where this ServerUID emerges
from the server config as is and only needs a user tweak for
exceptional cases.
>> Instead, as I said, we should have a specific balancer directive
>> or option, complementing the Persist-related ones, that either
>> have the config line number included or not in the hash, ala
>> "IgnoreConfigFileChange" (or whatever)
>
> That's a new directive too, very specialized (though it's your point).
> Wouldn't it end up being declined to other modules one day or the other?
> But mainly, rather than ignoring config changes, I think mod_proxy_lb
> should ignore non-changes.
> And it already knows how to do that provided the vhost's balancer id
> does not change unnecessarily...
>
> It really matters for (graceful) restarts only, because otherwise we
> don't need to reuse names or slotmems (besides Mark's unusual issue,
> see [1] below).
> Why would one restart httpd if not to apply some configuration change?
> Whether this change relates to balancers or not is something the
> webmaster shouldn't care about (and figure out if
> IgnoreConfigFileChange should be on/off, depending on the kind of
> change, before the next restart).
>
>
> Regards,
> Yann.
>
> [1] AIUI, the issue Mark encounters with balancers is that the main
> process crashes because of third party modules, leaking IPC-SysV SHMs
> to the system (many balancers thus many leaks).
> Since his configuration file changes often, the SHMs are not cleaned
> up on the next startup either (their names changed with line numbers).
> So I proposed, as a *workaround* before the root cause if fixed in
> third party modules and to unblock his production, a temporary patch
> to remove line numbers from the computation of the SHMs names.
> But the ServerUID proposal is not meant to address Mark's unusual
> case, I mean the unusual case where the parent process crashes,
> because (frequent) restarts are not that unusual and should be fixed
> IMO.
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Feb 2, 2018 at 5:52 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> But we already have a unique ID in the form of
> the actual server struct itself, which contains
> all the data required to make a vhost "unique"
Sure, my patch or any other way to compute an automatic UID for a
vhost don't need anything else than the server_rec.
More exactly the proposed patch needs all the servers because it
doesn't take the line number into account, and nothing prevents
multiple exact same vhosts to be configured (totally useless, so an
implementation detail and not the point here).
>
> The only "issue" is what makes a host "unique"... IMO,
> in an environ where the definition line of the vhost changes
> between restarts is a "new" vhost and that was the assumption
> that the implementation was based on.
I don't deny it can be a valid assumption for some cases, but I
observe that it breaks others like "automatic vhost UID should remain
the same when some totally unrelated line is added before it".
>
> *But that assumption can more easily be changed by providing
> a work-around for that specific use case*
I'm not proposing this directive to address the mod_proxy_lb case, and
just gave it as an example for an in-tree case where it would be
useful (at least a new automatic UID is needed I think).
That case just reminded me that I had a patch somewhere for the
ServerUID, which is actually part of a larger one for a per-vhost
scoreboard/logio stuff which I already talked about some times ago
(with no apparent feedback/interest).
This larger patch really needs a vhost UID for stats to be relevent
days/month/years later, regardless of minor configuration changes,
even for things like vhost IP or port (which does not matter too much
in a local network, not to talk about line numbers...) or
ServerName/Alias (nothing mandates them for a vhost to work, e.g. IPs
only and ProxyPreserveHost).
Anyway, if you already have an UID in an User Interface to identify a
vhost, put it in ServerUID for httpd to use it where needed internally
and/or to "print" that UID anywhere you configured it to (e.g.
%{SERVER_UID} in access_log, headers, expr, ...).
At the end of the day, the webmaster really talks about the ServerName
or this "UID" to refer to a vhost (they can be and even usually are
the same modulo ports 80/443), but never about the IP or line number
it once had.
Sooo :) instead of proposing the change on how we can uniquely
identify a vhost for mod_proxy_lb only (in an automatic way or not), I
thought ServerUID could start its upstream life.
> instead of creating a
> whole new Directive for a "generic" server UID which:
>
> 1. Implies that the webmaster must know when and how that UID
> is used internally
How is it different than now, or than anything we could do
automatically/unconditionaly based on the server_rec?
This argument sounds to me like we really need the ServerUID, if the
webmaster really wants httpd to use an ID which does not depend on
whatever future change to the configuration, (s)he has to provide that
ID or know the internals.
The only requirement on the httpd side is that an given ServerUID is
unique (that should be documented, though the directive name not
really ambiguous). It shouldn't be too hard for the user, UUID or
alike.
> 2. Add more Directive cruft to an already bloated collection
> of directives.
That's a very valid concern! But I think my proposal is quite
orthogonal to the mod_proxy_lb issue, and it doesn't avoid us to find
one (or Stefan's multiple) good way to uniquely identify a vhost when
no ServerUID is specified.
>
> Instead, as I said, we should have a specific balancer directive
> or option, complementing the Persist-related ones, that either
> have the config line number included or not in the hash, ala
> "IgnoreConfigFileChange" (or whatever)
That's a new directive too, very specialized (though it's your point).
Wouldn't it end up being declined to other modules one day or the other?
But mainly, rather than ignoring config changes, I think mod_proxy_lb
should ignore non-changes.
And it already knows how to do that provided the vhost's balancer id
does not change unnecessarily...
It really matters for (graceful) restarts only, because otherwise we
don't need to reuse names or slotmems (besides Mark's unusual issue,
see [1] below).
Why would one restart httpd if not to apply some configuration change?
Whether this change relates to balancers or not is something the
webmaster shouldn't care about (and figure out if
IgnoreConfigFileChange should be on/off, depending on the kind of
change, before the next restart).
Regards,
Yann.
[1] AIUI, the issue Mark encounters with balancers is that the main
process crashes because of third party modules, leaking IPC-SysV SHMs
to the system (many balancers thus many leaks).
Since his configuration file changes often, the SHMs are not cleaned
up on the next startup either (their names changed with line numbers).
So I proposed, as a *workaround* before the root cause if fixed in
third party modules and to unblock his production, a temporary patch
to remove line numbers from the computation of the SHMs names.
But the ServerUID proposal is not meant to address Mark's unusual
case, I mean the unusual case where the parent process crashes,
because (frequent) restarts are not that unusual and should be fixed
IMO.
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
But we already have a unique ID in the form of
the actual server struct itself, which contains
all the data required to make a vhost "unique"
The only "issue" is what makes a host "unique"... IMO,
in an environ where the definition line of the vhost changes
between restarts is a "new" vhost and that was the assumption
that the implementation was based on.
*But that assumption can more easily be changed by providing
a work-around for that specific use case* instead of creating a
whole new Directive for a "generic" server UID which:
1. Implies that the webmaster must know when and how that UID
is used internally
2. Add more Directive cruft to an already bloated collection
of directives.
Instead, as I said, we should have a specific balancer directive
or option, complementing the Persist-related ones, that either
have the config line number included or not in the hash, ala
"IgnoreConfigFileChange" (or whatever)
> On Feb 2, 2018, at 9:52 AM, Yann Ylavic <yl...@gmail.com> wrote:
>
> On Fri, Feb 2, 2018 at 3:50 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Fri, Feb 2, 2018 at 3:44 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>>
>>>
>>>> Am 02.02.2018 um 15:42 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>
>>>> On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
>>>> <ru...@vodafone.com> wrote:>
>>>>>
>>>>>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>>>>>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>>>>>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>>>>>> directive
>>>>>>
>>>>>> Why? If it is designed to not change between restarts then there
>>>>>> are much easier ways to be unique, which it kinda already is,
>>>>>> considering the actual structs being used.
>>>>
>>>> I don't know what "easier ways" you are thinking about, the one
>>>> proposed here is not that complicated IMO.
>>>> In any case the method we are currently using in mod_proxy_lb *is*
>>>> changing accross restarts, mainly because of the line number.
>>>> What if you add/remove a line before the <VirtualHost>?
>>>> At least for graceful restarts, I think it shall not change, SHMs
>>>> should be reused.
>>>
>>> Is it a hash across the config record of a server what would give
>>> the desired behaviour?
>>
>> Yes, a hash using the minimal (IP[:port])* + ServerName, which is what
>> matters to select a vhost in the first place (maybe it's missing
>> ServerAlias), and what's done in this patch.
>> Should two vhosts have the same hash, only the first one will ever
>> handle requests.
>
> But I still think that the ServerUID directive is useful because the
> above are not immutable for the lifetime of a vhost either.
> That would address all needs, IMO.
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Feb 2, 2018 at 3:50 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Fri, Feb 2, 2018 at 3:44 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>>
>>> Am 02.02.2018 um 15:42 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>> On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
>>> <ru...@vodafone.com> wrote:>
>>>>
>>>>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>>>>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>>>>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>>>>> directive
>>>>>
>>>>> Why? If it is designed to not change between restarts then there
>>>>> are much easier ways to be unique, which it kinda already is,
>>>>> considering the actual structs being used.
>>>
>>> I don't know what "easier ways" you are thinking about, the one
>>> proposed here is not that complicated IMO.
>>> In any case the method we are currently using in mod_proxy_lb *is*
>>> changing accross restarts, mainly because of the line number.
>>> What if you add/remove a line before the <VirtualHost>?
>>> At least for graceful restarts, I think it shall not change, SHMs
>>> should be reused.
>>
>> Is it a hash across the config record of a server what would give
>> the desired behaviour?
>
> Yes, a hash using the minimal (IP[:port])* + ServerName, which is what
> matters to select a vhost in the first place (maybe it's missing
> ServerAlias), and what's done in this patch.
> Should two vhosts have the same hash, only the first one will ever
> handle requests.
But I still think that the ServerUID directive is useful because the
above are not immutable for the lifetime of a vhost either.
That would address all needs, IMO.
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 02.02.2018 um 16:43 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Feb 2, 2018 at 4:28 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>>> Am 02.02.2018 um 15:50 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>>> Is it a hash across the config record of a server what would give
>>>> the desired behaviour?
>>>
>>> Yes, a hash using the minimal (IP[:port])* + ServerName, which is what
>>> matters to select a vhost in the first place (maybe it's missing
>>> ServerAlias), and what's done in this patch.
>>> Should two vhosts have the same hash, only the first one will ever
>>> handle requests.
>>
>> So, for my understanding, what we are looking for it:
>>
>> const char * ap_get_server_id(server_rec *s);
>
> Yes, mainly, which:
> {
> return r->server_id;
> }
> :)
>
>>
>> that gives a string usable as a file name which does not change
>> unless the given server has "changed"? And the distinction of
>> what exactly is the "change", we probably can argue millenia
>> about?
>
> Precisely, the admin knows..
> That doesn't prevent us from having a beer and talk about when he/she
> doesn't care and simply wants it to work ;)
I need to ponder this with a nice bottle of wine this evening...maybe.
My gut feeling is that, by pure numbers, most users want Apache to
serve domains one next to the other. They do not want overlaps, they
want http: and https: for the same domain to show the same resource (or
have the former redirect to the later).
I think they would like Apache to refuse a configuration where this
is not somehow true (e.g. ServerAlias/Name overlapping, a Location
slipped into one vhost of a domain, but not the other). The would
like Apache to answer 404 for any domain that is not configured.
Then we have fewer, but possibly larger and "more interesting" setups
where people use the config possibilities up to the max, know which
virtual host a fallback for unknown domains is, etc. etc. etc.
How far would come in satisfying all by having a core directive
DomainScope global | ports | sockets | free
with meaning:
free: as is now
sockets: domain settings are unique for each address+port (can be same for different addresses),
e.g. ServerName/-Alias props can differ on each listening socket
ports: domain settings are unique for each port number (irregardless of listening address)
e.g. ServerName/-Alias props can differ on other ports
global: domain settings are unique for all ports and addresses,
e.g. ServerName/-Alias props are the same everywhere in this server
?
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Feb 2, 2018 at 4:28 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
>
>> Am 02.02.2018 um 15:50 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>>> Is it a hash across the config record of a server what would give
>>> the desired behaviour?
>>
>> Yes, a hash using the minimal (IP[:port])* + ServerName, which is what
>> matters to select a vhost in the first place (maybe it's missing
>> ServerAlias), and what's done in this patch.
>> Should two vhosts have the same hash, only the first one will ever
>> handle requests.
>
> So, for my understanding, what we are looking for it:
>
> const char * ap_get_server_id(server_rec *s);
Yes, mainly, which:
{
return r->server_id;
}
:)
>
> that gives a string usable as a file name which does not change
> unless the given server has "changed"? And the distinction of
> what exactly is the "change", we probably can argue millenia
> about?
Precisely, the admin knows..
That doesn't prevent us from having a beer and talk about when he/she
doesn't care and simply wants it to work ;)
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 02.02.2018 um 15:50 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Feb 2, 2018 at 3:44 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>>
>>> Am 02.02.2018 um 15:42 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>> On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
>>> <ru...@vodafone.com> wrote:>
>>>>
>>>>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>>>>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>>>>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>>>>> directive
>>>>>
>>>>> Why? If it is designed to not change between restarts then there
>>>>> are much easier ways to be unique, which it kinda already is,
>>>>> considering the actual structs being used.
>>>
>>> I don't know what "easier ways" you are thinking about, the one
>>> proposed here is not that complicated IMO.
>>> In any case the method we are currently using in mod_proxy_lb *is*
>>> changing accross restarts, mainly because of the line number.
>>> What if you add/remove a line before the <VirtualHost>?
>>> At least for graceful restarts, I think it shall not change, SHMs
>>> should be reused.
>>
>> Is it a hash across the config record of a server what would give
>> the desired behaviour?
>
> Yes, a hash using the minimal (IP[:port])* + ServerName, which is what
> matters to select a vhost in the first place (maybe it's missing
> ServerAlias), and what's done in this patch.
> Should two vhosts have the same hash, only the first one will ever
> handle requests.
So, for my understanding, what we are looking for it:
const char * ap_get_server_id(server_rec *s);
that gives a string usable as a file name which does not change
unless the given server has "changed"? And the distinction of
what exactly is the "change", we probably can argue millenia
about?
-Stefan
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Feb 2, 2018 at 3:44 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
>
>
>> Am 02.02.2018 um 15:42 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
>> <ru...@vodafone.com> wrote:>
>>>
>>>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>>>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>>>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>>>> directive
>>>>
>>>> Why? If it is designed to not change between restarts then there
>>>> are much easier ways to be unique, which it kinda already is,
>>>> considering the actual structs being used.
>>
>> I don't know what "easier ways" you are thinking about, the one
>> proposed here is not that complicated IMO.
>> In any case the method we are currently using in mod_proxy_lb *is*
>> changing accross restarts, mainly because of the line number.
>> What if you add/remove a line before the <VirtualHost>?
>> At least for graceful restarts, I think it shall not change, SHMs
>> should be reused.
>
> Is it a hash across the config record of a server what would give
> the desired behaviour?
Yes, a hash using the minimal (IP[:port])* + ServerName, which is what
matters to select a vhost in the first place (maybe it's missing
ServerAlias), and what's done in this patch.
Should two vhosts have the same hash, only the first one will ever
handle requests.
Re: New ServerUID directive
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
> On 2 Feb 2018, at 15:44, Stefan Eissing <st...@greenbytes.de> wrote:
>
>
>
>> Am 02.02.2018 um 15:42 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
>> <ru...@vodafone.com> wrote:>
>>>
>>>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>>>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>>>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>>>> directive
>>>>
>>>> Why? If it is designed to not change between restarts then there
>>>> are much easier ways to be unique, which it kinda already is,
>>>> considering the actual structs being used.
>>
>> I don't know what "easier ways" you are thinking about, the one
>> proposed here is not that complicated IMO.
>> In any case the method we are currently using in mod_proxy_lb *is*
>> changing accross restarts, mainly because of the line number.
>> What if you add/remove a line before the <VirtualHost>?
>> At least for graceful restarts, I think it shall not change, SHMs
>> should be reused.
>
> Is it a hash across the config record of a server what would give
> the desired behaviour?
We have several internal modules which sort of need this. In each case we generate a sha256 and a UUID based on the FQDN or IP address and the (first) port number it listens on.
And we then log this with ‘info’ during startup; thus allowing an admin to at some later point add a ServerUUID to the list of ServerName, ServerAdmin, etc. as he or she sees fit.
As in practice - we found that in the few times when you need to change that UUID - you are in effect doing that as part of switching to a new origin server or something like that at protocol level with respect to things like locks, caches and other short-transactions/connection RESTy things surviving stuff.
That said - having this in the core and being able to routinely seed this extra into things like cookies is goodness. We do that a lot now - and it helps with simplifying compliance a lot.
Dw.
Re: New ServerUID directive
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 02.02.2018 um 15:42 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
> <ru...@vodafone.com> wrote:>
>>
>>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>>> directive
>>>
>>> Why? If it is designed to not change between restarts then there
>>> are much easier ways to be unique, which it kinda already is,
>>> considering the actual structs being used.
>
> I don't know what "easier ways" you are thinking about, the one
> proposed here is not that complicated IMO.
> In any case the method we are currently using in mod_proxy_lb *is*
> changing accross restarts, mainly because of the line number.
> What if you add/remove a line before the <VirtualHost>?
> At least for graceful restarts, I think it shall not change, SHMs
> should be reused.
Is it a hash across the config record of a server what would give
the desired behaviour?
Re: New ServerUID directive
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Feb 2, 2018 at 3:25 PM, Plüm, Rüdiger, Vodafone Group
<ru...@vodafone.com> wrote:>
>
>> -----Ursprüngliche Nachricht----- Von: Jim Jagielski
>> [mailto:jim@jaguNET.com] Gesendet: Freitag, 2. Februar 2018 15:15
>> An: httpd <de...@httpd.apache.org> Betreff: Re: New ServerUID
>> directive
>>
>> Why? If it is designed to not change between restarts then there
>> are much easier ways to be unique, which it kinda already is,
>> considering the actual structs being used.
I don't know what "easier ways" you are thinking about, the one
proposed here is not that complicated IMO.
In any case the method we are currently using in mod_proxy_lb *is*
changing accross restarts, mainly because of the line number.
What if you add/remove a line before the <VirtualHost>?
At least for graceful restarts, I think it shall not change, SHMs
should be reused.
>>
>> Also, this seems like unnecessary admin overhead for the
>> webmaster... if there is a need for such an ID, httpd should
>> provide for it automagically and not require users to have to
>> config one. It seems like config-file fluff to me, IMO.
>
> +1. If the current ID used is not meeting our requirements, my first
> thought would be if we could improve it to meet them.
I'm fine with removing the directive and let the automatic UID be
generated at startup, the thing is already there in this patch.
The directive simply helps for the case when the admin knows what to
do, and for instance wants the vhosts to remain the same even if an IP
address or a ServerName is changed.
Regards,
Yann.
AW: New ServerUID directive
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski [mailto:jim@jaguNET.com]
> Gesendet: Freitag, 2. Februar 2018 15:15
> An: httpd <de...@httpd.apache.org>
> Betreff: Re: New ServerUID directive
>
> Why? If it is designed to not change between restarts then
> there are much easier ways to be unique, which it kinda
> already is, considering the actual structs being used.
>
> Also, this seems like unnecessary admin overhead for the
> webmaster... if there is a need for such an ID, httpd should
> provide for it automagically and not require users to have
> to config one. It seems like config-file fluff to me, IMO.
+1. If the current ID used is not meeting our requirements, my first thought would be if we could improve it
to meet them.
Regards
Rüdiger
Re: New ServerUID directive
Posted by Jim Jagielski <ji...@jaguNET.com>.
Why? If it is designed to not change between restarts then
there are much easier ways to be unique, which it kinda
already is, considering the actual structs being used.
Also, this seems like unnecessary admin overhead for the
webmaster... if there is a need for such an ID, httpd should
provide for it automagically and not require users to have
to config one. It seems like config-file fluff to me, IMO.
> On Feb 1, 2018, at 12:54 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> I have this patch (attached) floating around that allows users to
> configure a *fixed* UID for each vhost.
>
> There are several places where we need an ID for vhosts, and where we
> compute one based on ServerName/Port, addresses, configuration path
> and line numbers, ...
> This UID could be used instead, moreover since it has the property to
> not change on (re)starts we can also use it for things bound to a
> vhost regardless of the startup and unrelated configuration changes
> (the attached patch uses it for SHMs in mod_proxy_balancer, as an
> example).
>
> If no ServerUID is configured, I _think_ we can compute one too,
> better than the one usually computed in our code since it won't change
> unless address(es)/port(s) or ServerName of the vhost changes (which
> is not a "light" change anyway).
> In any case the patch also handles collisions, if ever...
> So for this what the patch does is (with rationale in comment):
>
> + int i, *num;
> + apr_hash_t *servers_uids = apr_hash_make(p);
> []
> + if (!s->server_uid) {
> + server_addr_rec *addr;
> + apr_md5_ctx_t md5_ctx;
> + unsigned char md5[APR_MD5_DIGESTSIZE];
> +
> + /* Assumes the unique identifier of a vhost is its address(es)
> + * plus the ServerName:Port. Should two or more vhosts have this
> + * same identifier, the first one would always be elected to
> + * handle the requests, so this shouldn't be an issue...
> + */
> + apr_md5_init(&md5_ctx);
> + for (addr = s->addrs; addr; addr = addr->next) {
> + char host_ip[64]; /* for any IPv[46] string */
> + apr_sockaddr_ip_getbuf(host_ip, sizeof host_ip,
> + addr->host_addr);
> + apr_md5_update(&md5_ctx, (unsigned char *)host_ip,
> + strlen(host_ip));
> + apr_md5_update(&md5_ctx, (unsigned char *)&addr->host_port,
> + sizeof(addr->host_port));
> + }
> + apr_md5_update(&md5_ctx, (unsigned char *)s->server_hostname,
> + strlen(s->server_hostname));
> + apr_md5_update(&md5_ctx, (unsigned char *)&s->port,
> + sizeof(s->port));
> + apr_md5_final(md5, &md5_ctx);
> +
> + s->server_uid = apr_pescape_hex(p, md5, sizeof md5, 0);
> + }
> + /* Handle collisions, that's Unique ID! */
> + num = apr_hash_get(servers_uids, s->server_uid,
> + APR_HASH_KEY_STRING);
> + if (num) {
> + ++*num;
> + s->server_uid = apr_psprintf(p, "%s_%i", s->server_uid, *num);
> + }
> + else {
> + num = apr_pcalloc(p, sizeof *num);
> + apr_hash_set(servers_uids, s->server_uid,
> + APR_HASH_KEY_STRING, num);
> + }
> (Rest attached)
>
> WDYT?
> <server_uid.patch>
Re: New ServerUID directive
Posted by Exonetric <ma...@exonetric.com>.
On 1 Feb 2018, at 17:54, Yann Ylavic <yl...@gmail.com> wrote:
>
> I have this patch (attached) floating around that allows users to
> configure a *fixed* UID for each vhost.
>
> There are several places where we need an ID for vhosts, and where we
> compute one based on ServerName/Port, addresses, configuration path
> and line numbers, ...
> This UID could be used instead, moreover since it has the property to
> not change on (re)starts we can also use it for things bound to a
> vhost regardless of the startup and unrelated configuration changes
> (the attached patch uses it for SHMs in mod_proxy_balancer, as an
> example).
>
> If no ServerUID is configured, I _think_ we can compute one too,
> better than the one usually computed in our code since it won't change
> unless address(es)/port(s) or ServerName of the vhost changes (which
> is not a "light" change anyway).
> In any case the patch also handles collisions, if ever...
> So for this what the patch does is (with rationale in comment):
>
> + int i, *num;
> + apr_hash_t *servers_uids = apr_hash_make(p);
> []
> + if (!s->server_uid) {
> + server_addr_rec *addr;
> + apr_md5_ctx_t md5_ctx;
> + unsigned char md5[APR_MD5_DIGESTSIZE];
> +
> + /* Assumes the unique identifier of a vhost is its address(es)
> + * plus the ServerName:Port. Should two or more vhosts have this
> + * same identifier, the first one would always be elected to
> + * handle the requests, so this shouldn't be an issue...
> + */
> + apr_md5_init(&md5_ctx);
> + for (addr = s->addrs; addr; addr = addr->next) {
> + char host_ip[64]; /* for any IPv[46] string */
> + apr_sockaddr_ip_getbuf(host_ip, sizeof host_ip,
> + addr->host_addr);
> + apr_md5_update(&md5_ctx, (unsigned char *)host_ip,
> + strlen(host_ip));
> + apr_md5_update(&md5_ctx, (unsigned char *)&addr->host_port,
> + sizeof(addr->host_port));
> + }
> + apr_md5_update(&md5_ctx, (unsigned char *)s->server_hostname,
> + strlen(s->server_hostname));
> + apr_md5_update(&md5_ctx, (unsigned char *)&s->port,
> + sizeof(s->port));
> + apr_md5_final(md5, &md5_ctx);
> +
> + s->server_uid = apr_pescape_hex(p, md5, sizeof md5, 0);
> + }
> + /* Handle collisions, that's Unique ID! */
> + num = apr_hash_get(servers_uids, s->server_uid,
> + APR_HASH_KEY_STRING);
> + if (num) {
> + ++*num;
> + s->server_uid = apr_psprintf(p, "%s_%i", s->server_uid, *num);
> + }
> + else {
> + num = apr_pcalloc(p, sizeof *num);
> + apr_hash_set(servers_uids, s->server_uid,
> + APR_HASH_KEY_STRING, num);
> + }
> (Rest attached)
>
> WDYT?
I am surprised httpd doesn’t already have this, considering the fundamental role of the virtualhost container.
- Mark
Re: New ServerUID directive
Posted by "Helmut K. C. Tessarek" <te...@evermeet.cx>.
On 2018-02-01 12:54, Yann Ylavic wrote:
> I have this patch (attached) floating around that allows users to
> configure a *fixed* UID for each vhost.
What do you mean by *fixed*?
I thought the virtual host itself already has to be unique. As far as I
know, I can't have two virtual hosts with the same hostname and port.
So why is it necessary to create another unique id?
Cheers,
K. C.
--
regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944
/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/