[users@httpd] How to Proxy and Rewrite URLs and responses with multiple instances to replace in URL

["mmccarthy@tribloom.com" <mm...@tribloom.com>]

Hello.

I have been attempting to restrict access to Kibana/ElasticSearch using 
Apache 2.4. Kibana, ElasticSearch, and Apache are all on the same 
server. Kibana is on port 9292, ElasticSearch is on port 9200, Apache is 
on 80.

My approach has been to proxy request through Apache and rewrite URLs 
based on the current (basic) authenticated users. The URL's to 
ElasticSearch will be modified with the current user, which matches an 
index in ElasticSearch. There are 2 cases of URL that I need to deal with

1. http://54.68.74.245/logstash-2014.07.04,logstash-2014.07.05/_aliases
2. http://54.68.74.245/logstash-2014.07.04/_aliases

The 2nd is rather easy and is just a simple RewriteRule:
# If REMOTE_USER exists
RewriteCond %{LA-U:REMOTE_USER} !^$
# If the request is not of the form of URL 1
RewriteCond %{REQUEST_URI} !^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+),(.*)$
# Find and replace "logstash" with "logstash-<REMOTE_USER>"
RewriteRule ^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+)(.*)$ 
http://127.0.0.1:9200/$1logstash-%{LA-U:REMOTE_USER}-$2$3 [P]

Note that this rewrite proxies the request to the ElasticSearch server. 
I have a ProxyPassReverse setting for both ElasticSearch and Kibana and 
it is working as expected.

The 1st type of URL is much more difficult. There is an arbitrary number 
of indexes on the URL. My approach was to use a RewriteRule with the [N] 
(next) flag to "loop" and replace all occurrences in the URL:
RewriteCond %{LA-U:REMOTE_USER} !^$
RewriteRule ^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+)(.*)$ 
/$1logstash-%{LA-U:REMOTE_USER}-$2$3 [N]

This actually works as expected also. The side effect is that the 
request becomes a subrequest and doesn't get proxied, which means that 
it looks to the local filesystem instead of the ElasticSearch server:
RewriteCond %{LA-U:REMOTE_USER} !^$
RewriteCond %{REQUEST_URI} ^/(.*)logstash(.*)$
RewriteRule (.*) http://127.0.0.1:9200$1 [P]

I have used a redirect ([R]) instead of a proxy ([P]) which works 
perfectly. My problem is that I also need to modify the responses from 
ElasticSearch, which doesn't work with redirects but does work with 
proxies. The response is JSON and contains the name of the index in the 
URL i.e. "logstash-<REMOTE_USER>-2014.07.04". Kibana was looking for the 
index "logstash-2014.07.04" and the JSON response needs to be changed to 
match:

AddOutputFilterByType SUBSTITUTE application/json
Substitute s/logstash-.*?-([0-9]+.[0-9]+.[0-9]+)/logstash-$1

Again, this works fine for URLs of form 2. For form 1, if I use a URL 
rewrite, the Substitute is not called. If I use a proxy, the request is 
not forwarded.

Can anyone please offer any advice on how I may be able to solve this 
problem and pass the right user restricted URL to ElasticSearch and 
transform the response back so that Kibana is not aware of the change? 
To reiterate, I need to transform the URL from Kibana to ElasticSearch, 
then I need to modify the response to reverse that transformation.

I have looked into RewriteMap, but I am unsure if this would replace all 
occurrences and I would have to maintain a mapping with all 
REMOTE_USERs. I've looked at using a RewriteMap with a program, but then 
I couldn't pass the REMOTE_USER parameter. I haven't been able to figure 
out a way to make a RewriteRule that replaces multiple occurrences in 
one pass, which if possible, would solve my problem. I have also tried 
searching the web, the #httpd IRC channel, and several days of 
experimentation and reading the documentation.

Thanks,

-- 
Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] How to Proxy and Rewrite URLs and responses with multiple instances to replace in URL

["mmccarthy@tribloom.com" <mm...@tribloom.com>]

I actually found a solution myself. I'm sure it is not very efficient 
but it seems to work. My solution is to force a redirect when using the 
[N] flag:

RewriteRule ^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+)(.*)$ 
/$1logstash-%{LA-U:REMOTE_USER}-$2$3 [N,R]

This avoids the subrequest problem with the proxy.

I am now running into a problem where long URLs are getting a "(36)File 
name too long" error. I understand that this is related to the 
filesystems max file name length, but it shouldn't apply to rewrites and 
proxies. Any ideas on how to get around this issue?

Thanks,
Michael
On 9/15/2014 10:07 AM, mmccarthy@tribloom.com wrote:
> Hello.
>
> I have been attempting to restrict access to Kibana/ElasticSearch 
> using Apache 2.4. Kibana, ElasticSearch, and Apache are all on the 
> same server. Kibana is on port 9292, ElasticSearch is on port 9200, 
> Apache is on 80.
>
> My approach has been to proxy request through Apache and rewrite URLs 
> based on the current (basic) authenticated users. The URL's to 
> ElasticSearch will be modified with the current user, which matches an 
> index in ElasticSearch. There are 2 cases of URL that I need to deal with
>
> 1. http://54.68.74.245/logstash-2014.07.04,logstash-2014.07.05/_aliases
> 2. http://54.68.74.245/logstash-2014.07.04/_aliases
>
> The 2nd is rather easy and is just a simple RewriteRule:
> # If REMOTE_USER exists
> RewriteCond %{LA-U:REMOTE_USER} !^$
> # If the request is not of the form of URL 1
> RewriteCond %{REQUEST_URI} !^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+),(.*)$
> # Find and replace "logstash" with "logstash-<REMOTE_USER>"
> RewriteRule ^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+)(.*)$ 
> http://127.0.0.1:9200/$1logstash-%{LA-U:REMOTE_USER}-$2$3 [P]
>
> Note that this rewrite proxies the request to the ElasticSearch 
> server. I have a ProxyPassReverse setting for both ElasticSearch and 
> Kibana and it is working as expected.
>
> The 1st type of URL is much more difficult. There is an arbitrary 
> number of indexes on the URL. My approach was to use a RewriteRule 
> with the [N] (next) flag to "loop" and replace all occurrences in the 
> URL:
> RewriteCond %{LA-U:REMOTE_USER} !^$
> RewriteRule ^/(.*)logstash-([0-9]+.[0-9]+.[0-9]+)(.*)$ 
> /$1logstash-%{LA-U:REMOTE_USER}-$2$3 [N]
>
> This actually works as expected also. The side effect is that the 
> request becomes a subrequest and doesn't get proxied, which means that 
> it looks to the local filesystem instead of the ElasticSearch server:
> RewriteCond %{LA-U:REMOTE_USER} !^$
> RewriteCond %{REQUEST_URI} ^/(.*)logstash(.*)$
> RewriteRule (.*) http://127.0.0.1:9200$1 [P]
>
> I have used a redirect ([R]) instead of a proxy ([P]) which works 
> perfectly. My problem is that I also need to modify the responses from 
> ElasticSearch, which doesn't work with redirects but does work with 
> proxies. The response is JSON and contains the name of the index in 
> the URL i.e. "logstash-<REMOTE_USER>-2014.07.04". Kibana was looking 
> for the index "logstash-2014.07.04" and the JSON response needs to be 
> changed to match:
>
> AddOutputFilterByType SUBSTITUTE application/json
> Substitute s/logstash-.*?-([0-9]+.[0-9]+.[0-9]+)/logstash-$1
>
> Again, this works fine for URLs of form 2. For form 1, if I use a URL 
> rewrite, the Substitute is not called. If I use a proxy, the request 
> is not forwarded.
>
> Can anyone please offer any advice on how I may be able to solve this 
> problem and pass the right user restricted URL to ElasticSearch and 
> transform the response back so that Kibana is not aware of the change? 
> To reiterate, I need to transform the URL from Kibana to 
> ElasticSearch, then I need to modify the response to reverse that 
> transformation.
>
> I have looked into RewriteMap, but I am unsure if this would replace 
> all occurrences and I would have to maintain a mapping with all 
> REMOTE_USERs. I've looked at using a RewriteMap with a program, but 
> then I couldn't pass the REMOTE_USER parameter. I haven't been able to 
> figure out a way to make a RewriteRule that replaces multiple 
> occurrences in one pass, which if possible, would solve my problem. I 
> have also tried searching the web, the #httpd IRC channel, and several 
> days of experimentation and reading the documentation.
>
> Thanks,
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org