You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/11/09 01:38:23 UTC
[GitHub] [apisix] jagerzhang opened a new issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
jagerzhang opened a new issue #5451:
URL: https://github.com/apache/apisix/issues/5451
### Issue description
比如Nginx自带的响应内容替换库可以替换局部内容：http://nginx.org/en/docs/http/ngx_http_sub_module.html
```
sub_filter '<a href="http://127.0.0.1:8080/' '<a href="https://$host/';
```
比如还有个支持正则替换的库（印象中Openresty已默认支持）：ngx_http_substitutions_filter_module，可以通过正则表达式来替换内容：
```
subs_filter_types text/html text/css text/xml;
subs_filter st(\d*).example.com $1.example.com ir;
subs_filter a.example.com s.example.com;
subs_filter http://$host https://$host;
```
但是我看APISIX的 response-rewrite插件文档，好像只能支持完整替换，相当于直接将所有响应都换成插件设置的返回，并不支持部分内容替换，想问下这个插件是否支持部分替换或正则替换？如果不支持，APISIX有没有其他解决方案呢？
### Environment
- apisix version (cmd: `apisix version`): 2.10.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1065843697
> Well, personally I prefer to use an array of filters:
On this way, filter would take effect orderly by itself, right?
> I am afraid we can't do regex in the stream.
From [directives](http://nginx.org/en/docs/dirindex.html), Nginx's sub_filter only support in the http not stream.
So, I think support in the http by first, and support in the stream by second in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1074888092
@spacewander @tokers Please take a look on the second solution, I'm not sure if it's on the right way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-968581008
Does Customize Nginx Configuration work globally? I think it would be better to make the response-rewrite plugin support regular substitution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-968217059
@tokers 在APISIX配置中使用Nginx或openresty的配置，求一个指引文档，多谢~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1063801567
Above the conversation, I got these information:
Target: support specify content-type to filter, substitute response body by matching specific regex(fixed string as well) with replacement.
> regex mode supports to replace all or once, and case-sensitive or not
Solution:
Properties table
| name | description | type | default |
| ----- | ----------- | ---- | ------- |
| filter_type | which response content-type matched | array | "text/plain" |
| filter_pattern | pattern to match response body, could be regex | string | |
| filter_replace | substitution content | string | |
| filter_options | options for regex, detail see [here](https://github.com/openresty/lua-nginx-module#ngxrematch) | string | |
| filter_kind | regex or completely substitute mode, range [1,2] 1-completely 2-regex | integer | 1 |
| filter_scope | regex substitute once or global, range [1,2], 1-once 2-global | integer | 2 |
Main steps:
* step 1: check `conf.filter_kind`, equals to 1 then fallback to completely substitute, otherwise goto regex substitute(both in header_filter and body_filter).
* step 2: at `header_filter` phase, check match `conf.filter_type` or not, if not then return, otherwise then mark `ctx.response_rewrite_matched` true.
* step 3: at `body_filter` phase, `conf.filter_pattern` empty or if not `ctx.response_rewrite_matched` then return, otherwise `check conf.filter_scope`: scope 1 use `ngx.re.sub`, scope 2 use `ngx.re.gsub`.
* step 4. substitution:
```lua
local body = ngx.arg[1]
ngx.arg[1] = sub(conf.filter_pattern, conf.filter_replace, conf.filter_options)
```
* step 5: mark eof
```lua
ngx.arg[2] = true
```
Any advices welcome :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1075804014
We can use https://github.com/apache/apisix/blob/9e1988595c7145babfaa9de36030f6540686c866/apisix/core/response.lua#L177 to buffer the body
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-972464521
@tzssangglass @tokers 非常期待能支持一下这个Openresty默认支持的特性。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1064724614
Well, personally I prefer to use an array of filters:
```
"filters": [{
"expr": ... # if given, run the filter when it's evaluated to true. Can be used to support "content-type" and more.
"kind": "regex", # a string enum would be better
"scope": ...,
"pattern": ...,
...
}]
```
> equals to 1 then fallback to completely substitute, otherwise goto regex substitute(both in header_filter and body_filter).
"completely substitute" can be named as sub_filter substitute? As Nginx's sub_fitler supports variable, which isn't a plain substitute.
> local body = ngx.arg[1]
> ngx.arg[1] = sub(conf.filter_pattern, conf.filter_replace, conf.filter_options)
I am afraid we can't do regex in the stream. This requires the regex engine to have stream mode, for example, see https://github.com/google/re2/issues/127.
In fact, Nginx's sub_filter will buffer the data before substitution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander edited a comment on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
spacewander edited a comment on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1075804014
> step 3: at body_filter phase,
We can use https://github.com/apache/apisix/blob/9e1988595c7145babfaa9de36030f6540686c866/apisix/core/response.lua#L177 to buffer the body
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-968421737
> @tokers 在APISIX配置中使用Nginx或openresty的配置，求一个指引文档，多谢~
Please check out [Customize Nginx Configuration](http://apisix.apache.org/docs/apisix/customize-nginx-configuration/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-972465271
> > @tokers 在APISIX配置中使用Nginx或openresty的配置，求一个指引文档，多谢~
>
> Please check out [Customize Nginx Configuration](http://apisix.apache.org/docs/apisix/customize-nginx-configuration/).
粗略看了下，这样用需要改动配置文件，相当于每次改动都需要reload或重启，动作比较重。还是期待能在插件里面动态支持~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-968665320
> Does Customize Nginx Configuration work globally?
Yes, the range cross the route boundaries.
> I think it would be better to make the response-rewrite plugin support regular substitution.
That's might be a good idea, let's listen to more sounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1074917909
@kwanhur Could you also give some data examples according to the second solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1077539864
After have a look at existed codes, update solution v3 and example:
**Solution**
Properties table adds one `filters`
| name | description | type | default |
| ------- | ---------------------------------------- | ----- | ---------- |
| filters | filter expressions, include key attributes how to filter | array | `[]` empty |
filter key attributes
| name | description | type | default |
| ------- | ---------------------------------------- | ------ | --------- |
| ~~expr~~ | ~~match expression, adapters to [lua-resty-expr](https://github.com/api7/lua-resty-expr) , only on header~~ reuse `vars` | ~~map~~ | ~~{}~~ |
| ~~kind~~ | ~~regex or subfilter mode~~ | ~~string~~ | ~~subfilter~~ |
| scope | regex substitute once or global | string | "once" |
| ~~pattern~~regex | match pattern on response body, regex or fixed string both ok | string | "" |
| replace | regex substitution content | string | "" |
| options | regex options, detail see [here](https://github.com/openresty/lua-nginx-module#ngxrematch) | string | "" |
Main steps:
step 1: at `header_filter` phase, reuse `vars` to match the response headers.
step 2: at `body_filter` phase if `ctx.response_rewrite_matched` false then return, otherwise
* if `conf.filters` empty, backward to original body-replaced logic(backward compatibly).
* if `conf.filters` not empty, buffer response body with `core.response.hold_body_chunk(ctx)`, then foreach `filters` to filter, pseudo-code:
```lua
local body = core.response.hold_body_chunk(ctx)
for i, filter in ipairs(conf.filters) do
if filter.scope == "once" then
body = ngx.re.sub(body, filter.pattern, filter.replace, filter.options)
else
body = ngx.re.gsub(body, filter.pattern, filter.replace, filter.options)
end
end
ngx.arg[1] = body
```
**Example**
* with response code `200`, fixed string `hello` replace with `world`
```shell
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"methods": ["GET"],
"uri": "/test/index.html",
"plugins": {
"response-rewrite": {
"filters": [
{"scope": "global", "regex": "hello", "replace": "world", "options": "jio"}
]
"vars":[
[ "status","==",200 ]
]
}
}
}'
```
* with response content-type `text/plain`, regex `\d` replace with `*`
```shell
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"methods": ["GET"],
"uri": "/test/index.html",
"plugins": {
"response-rewrite": {
"filters": [
{"scope": "global", "regex": "\d", "replace": "*", "options": "jio"}
]
"vars":[
[ "content-type","==","text/plain"]
]
}
}
}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-964072942
@jagerzhang Yes, it cannot be supported by this plugin, but if you desire, just use the nginx or openresty way on APISIX, they're also feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1066083863
> On this way, filter would take effect orderly by itself, right?
Yes.
> I am afraid we can't do regex in the stream.
Actually, I mean the streaming processing (not the L4 proxy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1068909668
Solution updated.
Properties table adds one `filters`
| name | description | type | default |
| ------- | ---------------------------------------- | ----- | ---------- |
| filters | filter expressions, include key attributes how to filter | array | `[]` empty |
filter key attributes
| name | description | type | default |
| ------- | ---------------------------------------- | ------ | --------- |
| expr | match expression, adapters to [lua-resty-expr](https://github.com/api7/lua-resty-expr) , only on header | map | {} |
| kind | regex or subfilter mode | string | subfilter |
| options | regex options, detail see [here](https://github.com/openresty/lua-nginx-module#ngxrematch) | string | "" |
| scope | regex substitute once or global | string | "once" |
| replace | regex substitution content | string | "" |
| pattern | match pattern on response body, regex or fixed string both ok | string | "" |
Main steps:
step 1: at `header_filter` `body_filter` phase, if `filters` empty, then back to original body-replaced logic(backward compatibly); else then goto step 2.
step 2: at `header_filter` phase, foreach `filters` and eval `expr` ok or not.
- `ctx.filter_matches` to store every filter expr result[true|false]
- `ctx.filter_matched` to store total filters result(use to check response body need to filter or not)
step 3: at `body_filter` phase,
1. if `ctx.filter_matched` true and `ngx.arg[2]` false, then buffer the response body into `ctx.filter_body`.
2. if `ctx.filter_matched` true and `ngx.arg[2]` true, then foreach `filters` to filter, pseudo-code:
```lua
if ctx.filter_matched and ngx.arg[2] then
for i, filter in ipairs(conf.filters) do
if ctx.filter_matched[i] then
if filter.scope == "once" then
ctx.body = ngx.re.sub(ctx.body,filter.pattern, filter.replace, filter.options)
else
ctx.body = ngx.re.gsub(ctx.body,filter.pattern, filter.replace, filter.options)
end
end
end
ngx.arg[1] = ctx.body
end
```
Q1: it seems `filter.kind` is redundant, since step 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #5451: request help: Does response-rewrite plugin support partial or regular substitution
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #5451:
URL: https://github.com/apache/apisix/issues/5451#issuecomment-1078560446
LGTM, except for a small problem:
We need to check the result of `hold_body_chunk`:
https://github.com/apache/apisix/blob/6cd7b8d024d2b3da9db0a61dbff3b462059fa233/apisix/core/response.lua#L168-L173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org