[Proposal for trunk] Extending RewriteMap, Reusing ip network matches

[Rainer Jung <ra...@kippdata.de>]

1) Extending RewriteMap
=======================

I plan to extend key file handling in text file RewriteMap. At the 
moment keys are always matched as exact strings against the map. I want 
to add the ability to alternatively

a) match via regexp (and replace backreferences in the found values)
b) match via ip network notation like e.g. used in "Require ip ..."

This could be expressed in the configuration by adding another token 
after type:source, e.g. "exact", "regexp", "ip", where "exact" is the 
default.

Of course a) can be achieved without RewriteMap with a list of 
RewriteRules, but if you need a long list, the map is easier to maintain 
and understand, and it's also reloaded atomatically.

For b) there is no other solution at the moment.

2) Further reuse of ip network matches
======================================

a) RewriteCond

Furthermore I would like to extend RewriteCond with another CondPattern 
that matches ip addresses in the way noted under b) above, i.e. using ip 
address and network notation. It is very clumsy and not robust to try to 
do that via regexps. For that we would need to decide on another prefix 
character to use in the CondPattern, like e.g.

RewriteCond %{REMOTE_ADDR} @195.227.30.128/25

The preifx character "@" needs to be chosen carefully to prevent most 
existing RewriteCond lines from being broken (in case they already start 
with this prefix). Alternatively it could be done using a new flag.

b) SetEnvIf

The same extension could be added to SetEnvIf:

SetEnvIf Remote_Addr @195.227.30.128/25 isTrusted

Any comments? I'm willing to work on the implementation, but I'd like to 
hear, whether there are arguments against it.

Thanks!

Rainer

Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

[Rainer Jung <ra...@kippdata.de>]

On 18.12.2009 20:24, William A. Rowe Jr. wrote:
> Rainer Jung wrote:
>> On 17.12.2009 23:14, William A. Rowe Jr. wrote:
>>> Rainer Jung wrote:
>>>> 1) Extending RewriteMap
>>>> =======================
>>>>
>>>> I plan to extend key file handling in text file RewriteMap. At the
>>>> moment keys are always matched as exact strings against the map. I want
>>>> to add the ability to alternatively
>>>>
>>>> a) match via regexp (and replace backreferences in the found values)
>>>> b) match via ip network notation like e.g. used in "Require ip ..."
>>>>
>>>> This could be expressed in the configuration by adding another token
>>>> after type:source, e.g. "exact", "regexp", "ip", where "exact" is the
>>>> default.
>>>
>>> -1; you are going to completely hide the performance penalty from the
>>> user's attention.  In order to identify this properly, please extend with
>>> an alternate directive to activate this (slower) mapping, e.g.
>>> RewriteList
>>> or some such.
>>
>> Really? The feature is off by default, users need to activate by adding
>> one of the new match types. Old configs will not see a performance penalty.
>>
>> The penalty when using the new match type can easily be documented the
>> same way I would need to documnt RewriteList. I'm not sure whether
>> adding a new directive is the way to go, because the purpose of the list
>> is the same as of the map. Only the type of key matching changes. I'm
>> afraid it will lead to some confusion.
>
> Reviewers are more likely to catch a misconfig if you use RewriteList rather
> than some extra args.  This is why I deprecated<Container ~ foo>  so long
> ago, and replaced it with<ContainerMatch foo>.

Do you have an comment on the other part of the proposal, namely adding 
ip subnet match to RewriteCond and SetEnvIf by using a magic prefix 
character, like e.g.

RewriteCond %{REMOTE_ADDR} @195.227.30.128/25
SetEnvIf Remote_Addr @195.227.30.128/25 isTrusted

It seems you wouldn't like that as well in analogy to the "~" operator? 
Any other syntax idea how to express an ip subnet match instead of a regexp?

> RewriteList's will take only a subset of the storage types, right?

Yes, I was only planning to add it to txt (and rnd) types. "int" and 
"prg" don't make sense for this. Concerning the dbm storage type ...

> You aren't planning on using/reading large, nearly empty hash sets, are you?

... we would need to iterate over the entries during initial load to 
precompile the regexps and replace the dbm storage with an in-memory 
regexp array. Even if the dbm were not to large, that would pose the 
problems that the order of the dbm entries retrieved during iteration is 
not well-defined. So I'd say no, there's no good way of combining this 
with dbm.

Similarly for dbd we would need an additional ordered retrieval sql 
statement as part of the config and we would loose the ability to use 
data updated in the database without restart, because there are no file 
timestamps we can check to detect such data updates. So again I'd say 
there's not good way of implementing this for a dbd based RewriteMap.

Regards,

Rainer

Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

["William A. Rowe Jr." <wr...@rowe-clan.net>]

Rainer Jung wrote:
> On 17.12.2009 23:14, William A. Rowe Jr. wrote:
>> Rainer Jung wrote:
>>> 1) Extending RewriteMap
>>> =======================
>>>
>>> I plan to extend key file handling in text file RewriteMap. At the
>>> moment keys are always matched as exact strings against the map. I want
>>> to add the ability to alternatively
>>>
>>> a) match via regexp (and replace backreferences in the found values)
>>> b) match via ip network notation like e.g. used in "Require ip ..."
>>>
>>> This could be expressed in the configuration by adding another token
>>> after type:source, e.g. "exact", "regexp", "ip", where "exact" is the
>>> default.
>>
>> -1; you are going to completely hide the performance penalty from the
>> user's attention.  In order to identify this properly, please extend with
>> an alternate directive to activate this (slower) mapping, e.g.
>> RewriteList
>> or some such.
> 
> Really? The feature is off by default, users need to activate by adding
> one of the new match types. Old configs will not see a performance penalty.
> 
> The penalty when using the new match type can easily be documented the
> same way I would need to documnt RewriteList. I'm not sure whether
> adding a new directive is the way to go, because the purpose of the list
> is the same as of the map. Only the type of key matching changes. I'm
> afraid it will lead to some confusion.

Reviewers are more likely to catch a misconfig if you use RewriteList rather
than some extra args.  This is why I deprecated <Container ~ foo> so long
ago, and replaced it with <ContainerMatch foo>.

RewriteList's will take only a subset of the storage types, right?  You aren't
planning on using/reading large, nearly empty hash sets, are you?

Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

[Rainer Jung <ra...@kippdata.de>]

On 17.12.2009 23:14, William A. Rowe Jr. wrote:
> Rainer Jung wrote:
>> 1) Extending RewriteMap
>> =======================
>>
>> I plan to extend key file handling in text file RewriteMap. At the
>> moment keys are always matched as exact strings against the map. I want
>> to add the ability to alternatively
>>
>> a) match via regexp (and replace backreferences in the found values)
>> b) match via ip network notation like e.g. used in "Require ip ..."
>>
>> This could be expressed in the configuration by adding another token
>> after type:source, e.g. "exact", "regexp", "ip", where "exact" is the
>> default.
>
> -1; you are going to completely hide the performance penalty from the
> user's attention.  In order to identify this properly, please extend with
> an alternate directive to activate this (slower) mapping, e.g. RewriteList
> or some such.

Really? The feature is off by default, users need to activate by adding 
one of the new match types. Old configs will not see a performance penalty.

The penalty when using the new match type can easily be documented the 
same way I would need to documnt RewriteList. I'm not sure whether 
adding a new directive is the way to go, because the purpose of the list 
is the same as of the map. Only the type of key matching changes. I'm 
afraid it will lead to some confusion.

Regards,

Rainer

Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

["William A. Rowe Jr." <wr...@rowe-clan.net>]

Rainer Jung wrote:
> 1) Extending RewriteMap
> =======================
> 
> I plan to extend key file handling in text file RewriteMap. At the
> moment keys are always matched as exact strings against the map. I want
> to add the ability to alternatively
> 
> a) match via regexp (and replace backreferences in the found values)
> b) match via ip network notation like e.g. used in "Require ip ..."
> 
> This could be expressed in the configuration by adding another token
> after type:source, e.g. "exact", "regexp", "ip", where "exact" is the
> default.

-1; you are going to completely hide the performance penalty from the
user's attention.  In order to identify this properly, please extend with
an alternate directive to activate this (slower) mapping, e.g. RewriteList
or some such.

Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

[Rainer Jung <ra...@kippdata.de>]

On 17.12.2009 16:02, "Plüm, Rüdiger, VF-Group" wrote:
>
>
>> -----Original Message-----
>> From: Rainer Jung
>> Sent: Donnerstag, 17. Dezember 2009 15:56
>> To: dev@httpd.apache.org
>> Subject: Re: [Proposal for trunk] Extending RewriteMap,
>> Reusing ip network matches
>>
>> On 17.12.2009 14:32, "Plüm, Rüdiger, VF-Group" wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Rainer Jung
>>>> Sent: Donnerstag, 17. Dezember 2009 13:38
>>>> To: dev@httpd.apache.org
>>>> Subject: [Proposal for trunk] Extending RewriteMap, Reusing
>>>> ip network matches
>>>>
>>>> 1) Extending RewriteMap
>>>> =======================
>>>>
>>>> I plan to extend key file handling in text file RewriteMap. At the
>>>> moment keys are always matched as exact strings against the
>>>> map. I want
>>>> to add the ability to alternatively
>>>>
>>>> a) match via regexp (and replace backreferences in the
>> found values)
>>>
>>> Can you give an example for this?
>>> Where are these regexps? In the map file or in the
>> expression of the Map in the RewriteRule?
>>
>> The regular expressions are the first columns in the map file. The
>> actual keys searched for will be normal strings matched against those
>> regexps. The result returned is the second column with backreferences
>> resolved against the match from the first column and the key.
>
> I see some performance issue with this approach since each time one thing
> in the file changes *all* regexps need to be recompiled. If it is a large
> file that changes frequently this can eat up a considerable amount of cycles.

Right, I noticed that to, when I just now started to implement to learn 
more about possible problems. OTOH having the rules in one externally 
managed file can still be nice even without using the reloading ability. 
And if you do reload it, recompiling the regexps will likely be cheaper 
than a full server restart because of changed RewriteRules.

At the moment as long as a key hasn't been looked up before (i.e. is not 
cached), the file is opened for each lookup and read line by line. When 
using regexps it would be necessary to keep the compiled regexps in 
memory after the first file read (to avoid continuous recompilation).

So the first access to the file is expensive and we would also need some 
more memory. The impl wouldn't change the behaviour of the existing 
exact match case though.

Regards,

Rainer

RE: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

["Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>]

 

> -----Original Message-----
> From: Rainer Jung 
> Sent: Donnerstag, 17. Dezember 2009 15:56
> To: dev@httpd.apache.org
> Subject: Re: [Proposal for trunk] Extending RewriteMap, 
> Reusing ip network matches
> 
> On 17.12.2009 14:32, "Plüm, Rüdiger, VF-Group" wrote:
> >
> >
> >> -----Original Message-----
> >> From: Rainer Jung
> >> Sent: Donnerstag, 17. Dezember 2009 13:38
> >> To: dev@httpd.apache.org
> >> Subject: [Proposal for trunk] Extending RewriteMap, Reusing
> >> ip network matches
> >>
> >> 1) Extending RewriteMap
> >> =======================
> >>
> >> I plan to extend key file handling in text file RewriteMap. At the
> >> moment keys are always matched as exact strings against the
> >> map. I want
> >> to add the ability to alternatively
> >>
> >> a) match via regexp (and replace backreferences in the 
> found values)
> >
> > Can you give an example for this?
> > Where are these regexps? In the map file or in the 
> expression of the Map in the RewriteRule?
> 
> The regular expressions are the first columns in the map file. The 
> actual keys searched for will be normal strings matched against those 
> regexps. The result returned is the second column with backreferences 
> resolved against the match from the first column and the key.

I see some performance issue with this approach since each time one thing
in the file changes *all* regexps need to be recompiled. If it is a large
file that changes frequently this can eat up a considerable amount of cycles.

Regards

Rüdiger

Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

[Rainer Jung <ra...@kippdata.de>]

On 17.12.2009 14:32, "Plüm, Rüdiger, VF-Group" wrote:
>
>
>> -----Original Message-----
>> From: Rainer Jung
>> Sent: Donnerstag, 17. Dezember 2009 13:38
>> To: dev@httpd.apache.org
>> Subject: [Proposal for trunk] Extending RewriteMap, Reusing
>> ip network matches
>>
>> 1) Extending RewriteMap
>> =======================
>>
>> I plan to extend key file handling in text file RewriteMap. At the
>> moment keys are always matched as exact strings against the
>> map. I want
>> to add the ability to alternatively
>>
>> a) match via regexp (and replace backreferences in the found values)
>
> Can you give an example for this?
> Where are these regexps? In the map file or in the expression of the Map in the RewriteRule?

The regular expressions are the first columns in the map file. The 
actual keys searched for will be normal strings matched against those 
regexps. The result returned is the second column with backreferences 
resolved against the match from the first column and the key.

Regards,

Rainer

RE: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches

["Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>]

 

> -----Original Message-----
> From: Rainer Jung
> Sent: Donnerstag, 17. Dezember 2009 13:38
> To: dev@httpd.apache.org
> Subject: [Proposal for trunk] Extending RewriteMap, Reusing 
> ip network matches
> 
> 1) Extending RewriteMap
> =======================
> 
> I plan to extend key file handling in text file RewriteMap. At the 
> moment keys are always matched as exact strings against the 
> map. I want 
> to add the ability to alternatively
> 
> a) match via regexp (and replace backreferences in the found values)

Can you give an example for this?
Where are these regexps? In the map file or in the expression of the Map in the RewriteRule?

Regards

Rüdiger