You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "kingluo (via GitHub)" <gi...@apache.org> on 2023/04/03 09:41:06 UTC
[GitHub] [apisix] kingluo opened a new issue, #9226: bug: body-transformer must have larger priority than proxy-cache
kingluo opened a new issue, #9226:
URL: https://github.com/apache/apisix/issues/9226
### Current Behavior
If we use the memory cache type in proxy-cache, then the body-transformer will not work.
Because currently, the proxy-rewrite runs the body filter after the body-transformer, so it caches the transformed body but not the original one from upstream. At the next request, the cache will hit and provide the transformed body to the body-transformer, it failed.
### Expected Behavior
The proxy-cache should cache the original response from the upstream.
### Error Logs
2023/04/03 16:40:32 [error] 2705998#2705998: *279 [lua] body-transformer.lua:123: transform(): response body decode: /opt/apisix//deps/share/lua/5.1/xml2lua.lua:92: Err
or Parsing XML [char=1]
, body={"result": "hello"}, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
### Steps to Reproduce
```
c put /routes/1 -d '
{
"uri":"/capital",
"name":"TempConvertRoute",
"methods":[
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE",
"PURGE"
],
"plugins":{
"body-transformer":{
"request":{
"input_format":"json",
"template":"<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:web=\"http://www.oorsprong.org/websamples.countryinfo\"><soapenv:Header/><soapenv:Body><web:CapitalCity><web:sCountryISOCode>{{_escape_xml(country)}}</web:sCountryISOCode></web:CapitalCity></soapenv:Body></soapenv:Envelope>"
},
"response":{
"input_format":"xml",
"template":"{\"result\": {*_escape_json(Envelope.Body.CapitalCityResponse.CapitalCityResult)*}}"
}
},
"file-logger":{
"path":"logs/file.log"
},
"proxy-cache":{
"cache_strategy":"memory",
"cache_bypass":[
"$arg_bypass"
],
"cache_http_status":[
200
],
"cache_key":[
"$uri",
"-cache-id"
],
"cache_method":[
"POST"
],
"hide_cache_headers":true,
"no_cache":[
"$arg_test"
],
"cache_zone":"memory_cache"
},
"proxy-rewrite":{
"headers":{
"set":{
"Accept-Encoding":"identity",
"Content-Type":"text/xml"
}
},
"uri":"/websamples.countryinfo/CountryInfoService.wso"
},
"response-rewrite":{
"headers":{
"set":{
"Content-Type":"application/json"
}
}
}
},
"upstream":{
"nodes":{
"localhost:3000":1
}
},
"status":1
}
'
d post /capital -d '{"country":"foo"}' -v
```
### Environment
- APISIX version (run `apisix version`): master
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] monkeyDluffy6017 closed issue #9226: bug: body-transformer should have a smaller priority than proxy-cache
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 closed issue #9226: bug: body-transformer should have a smaller priority than proxy-cache
URL: https://github.com/apache/apisix/issues/9226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] leslie-tsang commented on issue #9226: bug: body-transformer must have larger priority than proxy-cache
Posted by "leslie-tsang (via GitHub)" <gi...@apache.org>.
leslie-tsang commented on issue #9226:
URL: https://github.com/apache/apisix/issues/9226#issuecomment-1495217798
> /assign
Thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9226: bug: body-transformer must have larger priority than proxy-cache
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9226:
URL: https://github.com/apache/apisix/issues/9226#issuecomment-1495206770
/assign
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9226: bug: body-transformer should have a smaller priority than proxy-cache
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9226:
URL: https://github.com/apache/apisix/issues/9226#issuecomment-1498360938
## try to fix
### without proxy-cache
```diff
- proxy-mirror # priority: 1010
- - proxy-cache # priority: 1009
+ # - proxy-cache # priority: 1009
- proxy-rewrite # priority: 1008
```
```bash
$ curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Wed, 05 Apr 2023 14:55:06 GMT
< Server: APISIX/3.2.0
<
* Connection #0 to host 127.0.0.1 left intact
{"result": "hello"}
$ curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Wed, 05 Apr 2023 14:55:08 GMT
< Server: APISIX/3.2.0
<
* Connection #0 to host 127.0.0.1 left intact
{"result": "hello"}
```
### proxy-cache with low priority
```diff
- body-transformer # priority: 1080
- proxy-mirror # priority: 1010
- proxy-cache # priority: 1009
- proxy-rewrite # priority: 1008
```
```bash
$ curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Apisix-Cache-Status: MISS
< Date: Wed, 05 Apr 2023 14:56:59 GMT
< Server: APISIX/3.2.0
<
* Connection #0 to host 127.0.0.1 left intact
{"result": "hello"}
$ curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Server: APISIX/3.2.0
< Date: Wed, 05 Apr 2023 14:56:59 GMT
< Age: 1
< Apisix-Cache-Status: HIT
<
* Connection #0 to host 127.0.0.1 left intact
```
### proxy-cache with high priority
```diff
plugins: # plugin list (sorted by priority)
+ - proxy-cache # priority: 24000
- real-ip # priority: 23000
- ai # priority: 22900
- client-control # priority: 22000
```
```shell
$ curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Apisix-Cache-Status: MISS
< Date: Wed, 05 Apr 2023 14:58:42 GMT
< Server: APISIX/3.2.0
<
* Connection #0 to host 127.0.0.1 left intact
{"result": "hello"}
$ curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Wed, 05 Apr 2023 14:58:42 GMT
< Server: APISIX/3.2.0
< Age: 1
< Apisix-Cache-Status: HIT
<
* Connection #0 to host 127.0.0.1 left intact
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9226: bug: body-transformer should have a smaller priority than proxy-cache
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9226:
URL: https://github.com/apache/apisix/issues/9226#issuecomment-1496977212
try reproduce
version info
```
the last commit of APISIX is `c86c05b72322a2267289c366eafc999107bb047c`
```
1. install the golang process
```
curl http://localhost:3000
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<SOAP-ENV:Body>
<ns2:CapitalCityResponse
xmlns:ns2="http://spring.io/guides/gs-producing-web-service">
<ns2:CapitalCityResult>hello</ns2:CapitalCityResult>
</ns2:CapitalCityResponse>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
```
2. confirm the router
```shell
curl -X GET http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'|jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1542 0 1542 0 0 149k 0 --:--:-- --:--:-- --:--:-- 150k
{
"value": {
"plugins": {
"proxy-cache": {
"cache_strategy": "memory",
"cache_bypass": [
"$arg_bypass"
],
"cache_http_status": [
200
],
"cache_key": [
"$uri",
"-cache-id"
],
"cache_ttl": 300,
"hide_cache_headers": true,
"no_cache": [
"$arg_test"
],
"cache_method": [
"POST"
],
"cache_zone": "memory_cache",
"cache_control": false
},
"body-transformer": {
"request": {
"template": "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:web=\"http://www.oorsprong.org/websamples.countryinfo\"><soapenv:Header/><soapenv:Body><web:CapitalCity><web:sCountryISOCode>{{_escape_xml(country)}}</web:sCountryISOCode></web:CapitalCity></soapenv:Body></soapenv:Envelope>",
"input_format": "json"
},
"response": {
"template": "{\"result\": {*_escape_json(Envelope.Body.CapitalCityResponse.CapitalCityResult)*}}",
"input_format": "xml"
}
},
"file-logger": {
"include_resp_body": false,
"path": "logs/file.log"
},
"proxy-rewrite": {
"uri": "/websamples.countryinfo/CountryInfoService.wso",
"headers": {
"set": {
"Accept-Encoding": "identity",
"Content-Type": "text/xml"
}
},
"use_real_request_uri_unsafe": false
},
"response-rewrite": {
"headers": {
"set": {
"Content-Type": "application/json"
}
},
"body_base64": false
}
},
"uri": "/capital",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE",
"PURGE"
],
"id": "1",
"priority": 0,
"status": 1,
"create_time": 1680670219,
"update_time": 1680674853,
"upstream": {
"nodes": {
"localhost:3000": 1
},
"type": "roundrobin",
"hash_on": "vars",
"scheme": "http",
"pass_host": "pass"
},
"name": "TempConvertRoute"
},
"createdIndex": 43,
"modifiedIndex": 477,
"key": "/apisix/routes/1"
}
```
3. however. I got a 5xx response
```
curl http://127.0.0.1:9080/capital -X POST -d '{"country":"foo"}' -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /capital HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 05 Apr 2023 06:18:56 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: close
< Server: APISIX/3.2.0
<
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
* Closing connection 0
```
the error msg as follow
```
2023/04/05 06:16:21 [error] 30746#30746: *17567 lua entry thread aborted: runtime error: /home/ec2-user/apisix/apisix/core/ctx.lua:309: variable "upstream_cache_key" not found for writing; maybe it is a built-in variable that is not changeable or you forgot to use "set $upstream_cache_key '';" in the config file to define it first
stack traceback:
coroutine 0:
[C]: in function 'error'
/usr/local/openresty/lualib/resty/core/var.lua:144: in function '__newindex'
/home/ec2-user/apisix/apisix/core/ctx.lua:309: in function '__newindex'
/home/ec2-user/apisix/apisix/plugins/proxy-cache/init.lua:151: in function 'phase_func'
/home/ec2-user/apisix/apisix/plugin.lua:1094: in function 'run_plugin'
/home/ec2-user/apisix/apisix/init.lua:666: in function 'http_access_phase'
access_by_lua(nginx.conf:318):2: in main chunk, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
2023/04/05 06:16:21 [error] 30746#30746: *17567 [lua] body-transformer.lua:123: transform(): response body decode: /home/ec2-user/apisix//deps/share/lua/5.1/xml2lua.lua:92: Unbalanced Tag (/body) [char=155]
, body=<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
2023/04/05 06:16:21 [error] 30746#30746: *17567 [lua] body-transformer.lua:202: phase_func(): failed to transform response body: <html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
2023/04/05 06:18:56 [error] 30746#30746: *21768 lua entry thread aborted: runtime error: /home/ec2-user/apisix/apisix/core/ctx.lua:309: variable "upstream_cache_key" not found for writing; maybe it is a built-in variable that is not changeable or you forgot to use "set $upstream_cache_key '';" in the config file to define it first
stack traceback:
coroutine 0:
[C]: in function 'error'
/usr/local/openresty/lualib/resty/core/var.lua:144: in function '__newindex'
/home/ec2-user/apisix/apisix/core/ctx.lua:309: in function '__newindex'
/home/ec2-user/apisix/apisix/plugins/proxy-cache/init.lua:151: in function 'phase_func'
/home/ec2-user/apisix/apisix/plugin.lua:1094: in function 'run_plugin'
/home/ec2-user/apisix/apisix/init.lua:666: in function 'http_access_phase'
access_by_lua(nginx.conf:318):2: in main chunk, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
2023/04/05 06:18:56 [error] 30746#30746: *21768 [lua] body-transformer.lua:123: transform(): response body decode: /home/ec2-user/apisix//deps/share/lua/5.1/xml2lua.lua:92: Unbalanced Tag (/body) [char=155]
, body=<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
2023/04/05 06:18:56 [error] 30746#30746: *21768 [lua] body-transformer.lua:202: phase_func(): failed to transform response body: <html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
, client: 127.0.0.1, server: _, request: "POST /capital HTTP/1.1", host: "127.0.0.1:9080"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org