You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Dave Wreski <dw...@guardiandigital.com.INVALID> on 2024/03/07 15:22:35 UTC

[users@httpd] RewriteMap and patterns

Hi,

I'm trying to use RewriteMap on a few thousand older articles to map 
them to their modern equivalent. The pattern matches, but then the 
redirect doesn't occur. The examples and apache docs say I should be 
using [PT] to pass-through, but it results in a 404:

[Thu Mar 07 09:56:47.696040 2024] [rewrite:trace5] [pid 95091:tid 95121] 
mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - - 
[linuxsecurity.com/sid#5590c7db70c8]
[rid#7f02f4016480/initial] map lookup OK: map=lsv2ids[txt] key=161567 -> 
val=/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago

[Thu Mar 07 09:56:47.696125 2024] [rewrite:trace2] [pid 95091:tid 95121] 
mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - - 
[linuxsecurity.com/sid#5590c7db70c8]
[rid#7f02f4016480/initial] rewrite '/content/view/161567' -> 
'/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago'

I have the following in the main VirtualHost section of my apache config 
for this domain:

   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [PT]

My map file simply contains this line:

161567 
/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago

This is a valid URL appearing as a 404:

68.195.111.33 - - [07/Mar/2024:10:13:59 -0500] "GET /content/view/161567 
HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 
X:"SAMEORIGIN" 1/1161742 1802/11477/2983 H:HTTP/1.1 
U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago

If I replace [PT] with [L,R=301] it successfully loads the destination 
link, but I'm concerned I may be creating an additional redirect. What's 
the proper way to do this in my case?


[users@httpd] unsubsribe

Posted by Dimitar Georgievski <mi...@gmail.com>.
>
>
>
>

Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
Hi,

>>> First make the proper redirections:
>>>
>>>   RewriteEngine on
>>>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>>>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]
>>>
>>> Define the fallback for requests not ending in PHP but for which I 
>>> hand the php handler to deal them.
>>>   FallbackResource /index.php
>>>
>>> Define the handler for php file requests
>>> <FilesMatch \.php>
>>>   SetHandler 
>>> "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
>>> </Filesmatch>
>
> Can I ask about this again? I've got this all set up with a static 
> map, but request for /content/view/NNNNN where NNNNN is some key that 
> is not mapped to a value results in a 400 error.
>
> Without this rewritemap/rewriterule, the request falls through to 
> Joomla where a redirection.php script is called that maps the key to 
> the corresponding value in a database.
>
> With the rewritemap/rewriterule, it appears it doesn't fall through to 
> the redirection.php script when there is no match.
>
> Maybe I'm missing the role of the FallbackResource here? However, it 
> doesn't appear it takes an argument - only an environment variable, so 
> not sure how that would apply.
>
> Ideas greatly appreciated.
>
If I could add to this, another idea I had was to run the 
redirection.php script directly from apache rather than in Joomla as a 
rewriterule. Something like this:

RewriteRule ^/content/view/([0-9]+)/? 
/redirection/index.php?type=view&ids=$1&uri=/content/view/$1 [L,R=301]

However, it's the same rewrite pattern as the above rewriterule, so not 
sure how that would work.

I should also mention the reason why I need both the static map and the 
redirection.php script is that I can't predict every conceivable pattern 
that could exist in a map. I was also thinking of putting the top 5000 
key/value in the map to be really quick, then let the redirection.php 
script process the remaining legacy 70,000 or so...




-- 

Profile Photo

	

Dave Wreski

Chief Executive Officer

	

Guardian Digital Logo <https://www.guardiandigital.com>

*We Make Email Safe For Business*

	

Guardian Digital Facebook <https://www.facebook.com/gdlinux> Guardian 
Digital Twitter <https://twitter.com/gdlinux> Guardian Digital Linkedin 
<https://www.linkedin.com/company/guardiandigital>

Phone Icon <tel:+16408009446>

	

(640)-800-9446 <tel:+16408009446>

	

Email Security

Web Icon <https://www.guardiandigital.com>

	

www.guardiandigital.com <https://www.guardiandigital.com>

	

Sender Fraud Protection

Email Icon <ma...@guardiandigital.com>

	

dwreski@guardiandigital.com

	

Training & Awareness

Location Icon <https://maps.app.goo.gl/aLZyUyCJznWoW1Jh8>

	

103 Godwin Ave, Suite 314, Midland Park, NJ 07432 
<https://maps.app.goo.gl/aLZyUyCJznWoW1Jh8>

Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
Hi,

>> First make the proper redirections:
>>
>>   RewriteEngine on
>>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]
>>
>> Define the fallback for requests not ending in PHP but for which I 
>> hand the php handler to deal them.
>>   FallbackResource /index.php
>>
>> Define the handler for php file requests
>> <FilesMatch \.php>
>>   SetHandler 
>> "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
>> </Filesmatch>

Can I ask about this again? I've got this all set up with a static map, 
but request for /content/view/NNNNN where NNNNN is some key that is not 
mapped to a value results in a 400 error.

Without this rewritemap/rewriterule, the request falls through to Joomla 
where a redirection.php script is called that maps the key to the 
corresponding value in a database.

With the rewritemap/rewriterule, it appears it doesn't fall through to 
the redirection.php script when there is no match.

Maybe I'm missing the role of the FallbackResource here? However, it 
doesn't appear it takes an argument - only an environment variable, so 
not sure how that would apply.

Ideas greatly appreciated.


Re: [users@httpd] RewriteMap and patterns

Posted by Yann Ylavic <yl...@gmail.com>.
On Sun, Mar 24, 2024 at 2:31 AM Dave Wreski
<dw...@guardiandigital.com.invalid> wrote:
>
> I'd like to be able to use the static map for the IDs I do know, and send any others to the redirect script directly from within my apache config directly. Is this possible?
>
> Something like this:
>
>   RewriteEngine on
>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>   RewriteRule "/content/view/([0-9]{6})/?" "${lsv2ids:$1}" [L]
>
>   RewriteRule "/content/view/([0-9]{6})/?" "https://linuxsecurity.com/redirection/index.php?type=view&ids=$1&uri=$1"

Possibly using the "DefaultValue" (i.e. not found) as described in [1]:

  RewriteRule "/content/view/([0-9]{6})/?"
"${lsv2ids:$1|https://linuxsecurity.com/redirection/index.php?type=view&ids=$1&uri=$1}"
[L]

[1] https://httpd.apache.org/docs/2.4/rewrite/rewritemap.html


Regards;
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
Hi,

> What I would do is RewriteRule with just [L] tag for internal 
> redirection or [R,L] if you don't mind the redirection to be external.
>
> Since I haven't seen the whole configuration my guess is you don't 
> have the proper handler defined for the resulting url and that's why 
> you get a 404, and in those cases a is when fallbackresource is ideal, 
> to hand those requests to where they must end.
>
> Generally when I use passthrough [PT] is because the end url will be 
> dealt with the appropiate handler, but you don't seem to have a rule 
> for the destination to be handled in php, so how come you expect 
> anything else other than a 404?
>
> That is briefly what I would do:
>
> First make the proper redirections:
>
>   RewriteEngine on
>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]
>
> Define the fallback for requests not ending in PHP but for which I 
> hand the php handler to deal them.
>   FallbackResource /index.php
>
> Define the handler for php file requests
> <FilesMatch \.php>
>   SetHandler 
> "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
> </Filesmatch>

I've got all this working. However, we currently have a PHP script 
that's used to catch requests that are not in our static map. It is 
spawned from Joomla. Normally, when a URL like the following is 
requested and no match is found in our static map:

https://linuxsecurity.com/content/view/127853

it is then passed onto the PHP script from within Joomla that sets the 
Location to something like:

https://linuxsecurity.com/redirection/index.php?type=view&ids=127853&uri=/content/view/127853/

The redirection.php script's function is to perform an SQL query that 
tries to identify the current name from that article ID and redirects 
the user to something like the following:

https://linuxsecurity.com/news/network-security/the-changing-security-threat-of-email-attacks

I'd like to be able to use the static map for the IDs I do know, and 
send any others to the redirect script directly from within my apache 
config directly. Is this possible?

Something like this:

   RewriteEngine on
   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
   RewriteRule "/content/view/([0-9]{6})/?" "${lsv2ids:$1}" [L]

   RewriteRule "/content/view/([0-9]{6})/?" 
"https://linuxsecurity.com/redirection/index.php?type=view&ids=$1&uri=$1"

I don't see how this would work, since both rules contain the same 
pattern. Ideas on how to go about this would be appreciated. Perhaps a 
RewriteMap prg rule?

   RewriteMap mapper "prg:/etc/httpd/conf.d/mapper.php"
   RewriteRule "/content/view/([0-9]{6})/?" "${mapper:$1}"

Thanks,
Dave




Re: [users@httpd] RewriteMap and patterns

Posted by Daniel Ferradal Márquez <df...@apache.org>.
On 12/03/2024 20:45, Dave Wreski wrote:

>>>
>> If I understood correctly you want the resulting path to be loaded by 
>> your php-fpm backend.
>>
>> Do you have a fallbackresource, such as:
>> "FallbackResource /index.php" or similar?
>>
>> If not, then you may want to add it.
> 
> Yes, that's correct, I'm using php-fpm. I don't currently have a 
> fallbackresource. I'll add it, if necessary, but I'd also like to 
> identify the actual source reason why it's happening in the first place. 
> How do I troubleshoot this further?
> 
> Thanks,
> Dave
> 
> 
> 
> 

What I would do is RewriteRule with just [L] tag for internal 
redirection or [R,L] if you don't mind the redirection to be external.

Since I haven't seen the whole configuration my guess is you don't have 
the proper handler defined for the resulting url and that's why you get 
a 404, and in those cases a is when fallbackresource is ideal, to hand 
those requests to where they must end.

Generally when I use passthrough [PT] is because the end url will be 
dealt with the appropiate handler, but you don't seem to have a rule for 
the destination to be handled in php, so how come you expect anything 
else other than a 404?

That is briefly what I would do:

First make the proper redirections:

   RewriteEngine on
   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]

Define the fallback for requests not ending in PHP but for which I hand 
the php handler to deal them.
   FallbackResource /index.php

Define the handler for php file requests
<FilesMatch \.php>
   SetHandler "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
</Filesmatch>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
On 3/11/24 4:42 PM, apmail-dferradal@apache.org wrote:
> On 09/03/2024 22:59, Dave Wreski wrote:
>> Hi Eric,
>>
>>> Might have to prefix with %{DOCUMENT_ROOT}
>>
>> I've tried variations of that, including the following, but no luck. 
>> They all fail with 400 or "invalid request"
>>
>> RewriteRule "/content/view/(.*)" "${lsv2ids:$1}"
>> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}" [PT]
>> RewriteRule "/content/view/(.*)" 
>> "https://linuxsecurity.com${lsv2ids:$1}" [PT]
>> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}"
>>
> If I understood correctly you want the resulting path to be loaded by 
> your php-fpm backend.
>
> Do you have a fallbackresource, such as:
> "FallbackResource /index.php" or similar?
>
> If not, then you may want to add it.

Yes, that's correct, I'm using php-fpm. I don't currently have a 
fallbackresource. I'll add it, if necessary, but I'd also like to 
identify the actual source reason why it's happening in the first place. 
How do I troubleshoot this further?

Thanks,
Dave




Re: [users@httpd] RewriteMap and patterns

Posted by ap...@apache.org.
On 09/03/2024 22:59, Dave Wreski wrote:
> Hi Eric,
> 
>> Might have to prefix with %{DOCUMENT_ROOT}
> 
> I've tried variations of that, including the following, but no luck. 
> They all fail with 400 or "invalid request"
> 
> RewriteRule "/content/view/(.*)" "${lsv2ids:$1}"
> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}" [PT]
> RewriteRule "/content/view/(.*)" 
> "https://linuxsecurity.com${lsv2ids:$1}" [PT]
> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}"
> 
> dave
> 
> 

If I understood correctly you want the resulting path to be loaded by 
your php-fpm backend.

Do you have a fallbackresource, such as:
"FallbackResource /index.php" or similar?

If not, then you may want to add it.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
Hi Eric,

> Might have to prefix with %{DOCUMENT_ROOT}

I've tried variations of that, including the following, but no luck. 
They all fail with 400 or "invalid request"

RewriteRule "/content/view/(.*)" "${lsv2ids:$1}"
RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}" [PT]
RewriteRule "/content/view/(.*)" 
"https://linuxsecurity.com${lsv2ids:$1}" [PT]
RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}"

dave


Re: [users@httpd] RewriteMap and patterns

Posted by Eric Covener <co...@gmail.com>.
Might have to prefix with %{DOCUMENT_ROOT}

On Sat, Mar 9, 2024 at 11:48 AM Eric Covener <co...@gmail.com> wrote:
>
> Try without [PT].
>
> On Sat, Mar 9, 2024 at 11:17 AM Dave Wreski
> <dw...@guardiandigital.com.invalid> wrote:
> >
> > Hi,
> >
> > I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
> > rewrites put the substitution) to setup the proxy sub-modules when
> > using SetHandler to configure proxying.
> >
> > I'm not sure I understand what that means - do you have a recommendation for how I should configure this instead?
> >
> > dave
> >
> >
> >
>
>
> --
> Eric Covener
> covener@gmail.com



-- 
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] RewriteMap and patterns

Posted by Eric Covener <co...@gmail.com>.
Try without [PT].

On Sat, Mar 9, 2024 at 11:17 AM Dave Wreski
<dw...@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
> rewrites put the substitution) to setup the proxy sub-modules when
> using SetHandler to configure proxying.
>
> I'm not sure I understand what that means - do you have a recommendation for how I should configure this instead?
>
> dave
>
>
>


-- 
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
Hi,

> I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
> rewrites put the substitution) to setup the proxy sub-modules when
> using SetHandler to configure proxying.

I'm not sure I understand what that means - do you have a recommendation 
for how I should configure this instead?

dave



Re: [users@httpd] RewriteMap and patterns

Posted by Eric Covener <co...@gmail.com>.
I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
rewrites put the substitution) to setup the proxy sub-modules when
using SetHandler to configure proxying.

On Fri, Mar 8, 2024 at 3:46 PM Dave Wreski
<dw...@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> What you have looks reasonable, PT should be URL-to-URL.
> You might try logging %f in your logformat.
>
> I've added %f and it just shows some of what looks like components from the 404 page, not the redirected page.
>
> What kind of handler is meant to handle these requests and how is it configured?
>
> Do you mean php-fpm?
>
> SetHandler "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
>
> This is the first log entry:
>
> f:proxy:fcgi://localhost/var/www/linuxsecurity.com/html/index.php 68.195.111.33 - - [08/Mar/2024:15:29:19 -0500] "GET /content/view/161567 HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" X:"SAMEORIGIN" 0/932130 1690/11576/2983 H:HTTP/1.1 U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago ls443 s:404 V:linuxsecurity.com v:linuxsecurity.com
>
> Thanks,
> Dave
>
>


-- 
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] RewriteMap and patterns

Posted by Dave Wreski <dw...@guardiandigital.com.INVALID>.
Hi,

> What you have looks reasonable, PT should be URL-to-URL.
> You might try logging %f in your logformat.

I've added %f and it just shows some of what looks like components from 
the 404 page, not the redirected page.

> What kind of handler is meant to handle these requests and how is it configured?

Do you mean php-fpm?

SetHandler "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"

This is the first log entry:

f:proxy:fcgi://localhost/var/www/linuxsecurity.com/html/index.php 
68.195.111.33 - - [08/Mar/2024:15:29:19 -0500] "GET /content/view/161567 
HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 
X:"SAMEORIGIN" 0/932130 1690/11576/2983 H:HTTP/1.1 
U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago 
ls443 s:404 V:linuxsecurity.com v:linuxsecurity.com

Thanks,
Dave


Re: [users@httpd] RewriteMap and patterns

Posted by Eric Covener <co...@gmail.com>.
What you have looks reasonable, PT should be URL-to-URL.
You might try logging %f in your logformat.

What kind of handler is meant to handle these requests and how is it configured?

On Thu, Mar 7, 2024 at 10:24 AM Dave Wreski
<dw...@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> I'm trying to use RewriteMap on a few thousand older articles to map them to their modern equivalent. The pattern matches, but then the redirect doesn't occur. The examples and apache docs say I should be using [PT] to pass-through, but it results in a 404:
>
> [Thu Mar 07 09:56:47.696040 2024] [rewrite:trace5] [pid 95091:tid 95121] mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - - [linuxsecurity.com/sid#5590c7db70c8]
> [rid#7f02f4016480/initial] map lookup OK: map=lsv2ids[txt] key=161567 -> val=/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
>
> [Thu Mar 07 09:56:47.696125 2024] [rewrite:trace2] [pid 95091:tid 95121] mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - - [linuxsecurity.com/sid#5590c7db70c8]
> [rid#7f02f4016480/initial] rewrite '/content/view/161567' -> '/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago'
>
> I have the following in the main VirtualHost section of my apache config for this domain:
>
>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [PT]
>
> My map file simply contains this line:
>
> 161567 /news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
>
> This is a valid URL appearing as a 404:
>
> 68.195.111.33 - - [07/Mar/2024:10:13:59 -0500] "GET /content/view/161567 HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" X:"SAMEORIGIN" 1/1161742 1802/11477/2983 H:HTTP/1.1 U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
>
> If I replace [PT] with [L,R=301] it successfully loads the destination link, but I'm concerned I may be creating an additional redirect. What's the proper way to do this in my case?
>
>


-- 
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org