You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/09/20 03:23:32 UTC

[GitHub] [apisix] soulbird opened a new pull request, #7947: feat(ext-plugin-post-resp): support get response body by extra_info

soulbird opened a new pull request, #7947:
URL: https://github.com/apache/apisix/pull/7947

   ### Description
   
   Provide the `ext-plugin-post-resp` plugin with the ability to get the response body, so that the External Plugin will be able to do some processing on the response body.
   As an additional feature, the `ext-plugin-post-resp` plugin also supports getting the request context in the response phase.
   
   Fixes # (issue)
   
   ### Checklist
   
   - [ ] I have explained the need for this PR and the problem it solves
   - [ ] I have explained the changes or the new features added to this PR
   - [ ] I have added tests corresponding to this change
   - [ ] I have updated the documentation to reflect this change
   - [ ] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first)
   
   <!--
   
   Note
   
   1. Mark the PR as draft until it's ready to be reviewed.
   2. Always add/update tests for any changes unless you have a good reason.
   3. Always update the documentation to reflect the changes made in the PR.
   4. Make a new commit to resolve conversations instead of `push -f`.
   5. To resolve merge conflicts, merge master instead of rebasing.
   6. Use "request review" to notify the reviewer after making changes.
   7. Only a reviewer can mark a conversation as resolved.
   
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] tzssangglass commented on pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on PR #7947:
URL: https://github.com/apache/apisix/pull/7947#issuecomment-1254481582

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] spacewander commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r977397141


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -20,6 +20,7 @@ repeat_each(1);
 no_long_string();
 no_root_location();
 no_shuffle();
+log_level("info");

Review Comment:
   The default level is already info? https://github.com/apache/apisix/blob/a47d05a9a32db1ba7de6bda68c67cf0008f8442c/t/APISIX.pm#L24



##########
apisix/plugins/ext-plugin/init.lua:
##########
@@ -304,7 +305,31 @@ local function handle_extra_info(ctx, input)
         if err then
             core.log.error("failed to read request body: ", err)
         end
-
+    elseif info_type == extra_info.RespBody then
+        local ext_res = ctx.runner_ext_response
+        if ext_res then
+            local info = req:Info()
+            local respbody_req = extra_info_respbody.New()
+            respbody_req:Init(info.byte, info.pos)
+
+            local chunks = {}
+            local err = helper.response_reader(ext_res.body_reader, function (chunk, chunks)
+                -- When the upstream response is chunked type,
+                -- we will receive the complete response body
+                -- before sending it to the runner program
+                -- to reduce the number of RCP calls.

Review Comment:
   ```suggestion
                   -- to reduce the number of RPC calls.
   ```



##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,156 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)

Review Comment:
   Let's add a test with empty response body



##########
apisix/plugins/ext-plugin/helper.lua:
##########
@@ -56,4 +56,27 @@ function _M.get_conf_token_cache_time()
 end
 
 
+function _M.response_reader(reader, callback, ...)
+    if not reader then
+        return "get response reader failed"
+    end
+
+    repeat
+        local chunk, read_err, cb_err
+        -- TODO: HEAD or 304

Review Comment:
   Could you use at least one sentence for this TODO? I can't recall why this comment is added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] soulbird commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
soulbird commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r975921673


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,161 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = nil}
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- error_log: failed to read response body: not exits
+
+
+
+=== TEST 6: add route with ext-plugin-post-resp
+--- config
+    location /t {
+        content_by_lua_block {
+            local json = require("toolkit.json")
+            local t = require("lib.test_admin")
+
+            local code, message, res = t.test('/apisix/admin/routes/1',
+                ngx.HTTP_PUT,
+                 [[{
+                    "uri": "/*",
+                    "plugins": {
+                        "ext-plugin-post-resp": {
+                        }
+                    },
+                    "upstream": {
+                        "nodes": {
+                            "127.0.0.1:1980": 1
+                        },
+                        "type": "roundrobin"
+                    }
+                }]]
+            )
+
+            if code >= 300 then
+                ngx.status = code
+                ngx.say(message)
+                return
+            end
+
+            ngx.say(message)
+        }
+    }
+--- response_body
+passed
+
+
+
+=== TEST 7: ask var
+--- request
+GET /hello?x=
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "var", name = "server_addr", result = "127.0.0.1"},
+                {type = "var", name = "remote_addr", result = "127.0.0.1"},
+                {type = "var", name = "route_id", result = "1"},
+                {type = "var", name = "arg_x", result = ""},
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- grep_error_log eval
+qr/send extra info req successfully/
+--- grep_error_log_out
+send extra info req successfully
+send extra info req successfully
+send extra info req successfully
+send extra info req successfully
+--- response_body
+hello world
+
+
+
+=== TEST 8: ask response body
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = "hello world\n"},
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- grep_error_log eval
+qr/send extra info req successfully/
+--- grep_error_log_out
+send extra info req successfully
+--- response_body
+hello world
+
+
+
+=== TEST 9: ask response body (chunked)
+--- request
+GET /hello_chunked
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = "hello world\n"},
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- grep_error_log eval
+qr/send extra info req successfully/
+--- grep_error_log_out
+send extra info req successfully
+--- response_body
+hello world
+
+
+
+=== TEST 10: ask request body (not exist)

Review Comment:
   I reused the [handle_extra_info](https://github.com/apache/apisix/blob/master/apisix/plugins/ext-plugin/init.lua#L283) function to get the response body, the request body is part of this function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] tzssangglass commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r975376842


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,161 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = nil}
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- error_log: failed to read response body: not exits
+
+
+
+=== TEST 6: add route with ext-plugin-post-resp
+--- config
+    location /t {
+        content_by_lua_block {
+            local json = require("toolkit.json")
+            local t = require("lib.test_admin")
+
+            local code, message, res = t.test('/apisix/admin/routes/1',
+                ngx.HTTP_PUT,
+                 [[{
+                    "uri": "/*",
+                    "plugins": {
+                        "ext-plugin-post-resp": {
+                        }
+                    },
+                    "upstream": {
+                        "nodes": {
+                            "127.0.0.1:1980": 1
+                        },
+                        "type": "roundrobin"
+                    }
+                }]]
+            )
+
+            if code >= 300 then
+                ngx.status = code
+                ngx.say(message)
+                return
+            end
+
+            ngx.say(message)
+        }
+    }
+--- response_body
+passed
+
+
+
+=== TEST 7: ask var
+--- request
+GET /hello?x=
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "var", name = "server_addr", result = "127.0.0.1"},
+                {type = "var", name = "remote_addr", result = "127.0.0.1"},
+                {type = "var", name = "route_id", result = "1"},
+                {type = "var", name = "arg_x", result = ""},
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- grep_error_log eval
+qr/send extra info req successfully/
+--- grep_error_log_out
+send extra info req successfully
+send extra info req successfully
+send extra info req successfully
+send extra info req successfully
+--- response_body
+hello world
+
+
+
+=== TEST 8: ask response body
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = "hello world\n"},
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- grep_error_log eval
+qr/send extra info req successfully/
+--- grep_error_log_out
+send extra info req successfully
+--- response_body
+hello world
+
+
+
+=== TEST 9: ask response body (chunked)
+--- request
+GET /hello_chunked
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = "hello world\n"},
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200
+--- grep_error_log eval
+qr/send extra info req successfully/
+--- grep_error_log_out
+send extra info req successfully
+--- response_body
+hello world
+
+
+
+=== TEST 10: ask request body (not exist)

Review Comment:
   why we check request body here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] tzssangglass commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r975373484


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,161 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = nil}
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200

Review Comment:
   would valid error_code by  default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] soulbird commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
soulbird commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r975921901


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,161 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = nil}
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200

Review Comment:
   Sure, I'll remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] soulbird commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
soulbird commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r977763053


##########
apisix/plugins/ext-plugin/helper.lua:
##########
@@ -56,4 +56,27 @@ function _M.get_conf_token_cache_time()
 end
 
 
+function _M.response_reader(reader, callback, ...)
+    if not reader then
+        return "get response reader failed"
+    end
+
+    repeat
+        local chunk, read_err, cb_err
+        -- TODO: HEAD or 304

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] spacewander merged pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
spacewander merged PR #7947:
URL: https://github.com/apache/apisix/pull/7947


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] tzssangglass commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r975373484


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,161 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)
+--- request
+GET /hello
+--- extra_stream_config
+    server {
+        listen unix:$TEST_NGINX_HTML_DIR/nginx.sock;
+
+        content_by_lua_block {
+            local ext = require("lib.ext-plugin")
+            local actions = {
+                {type = "respbody", result = nil}
+            }
+            ext.go({extra_info = actions})
+        }
+    }
+--- error_code: 200

Review Comment:
   would valid error_code == 200 by default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] soulbird commented on a diff in pull request #7947: feat(ext-plugin-post-resp): support get response body by extra_info

Posted by GitBox <gi...@apache.org>.
soulbird commented on code in PR #7947:
URL: https://github.com/apache/apisix/pull/7947#discussion_r977704185


##########
t/plugin/ext-plugin/extra-info.t:
##########
@@ -181,3 +182,156 @@ GET /hello
 --- error_code: 503
 --- error_log
 failed to receive RPC_HTTP_REQ_CALL: closed
+
+
+
+=== TEST 5: ask response body (not exist)

Review Comment:
   It doesn't make sense to add a test with an empty response body in the `ext-plugin-pre-req` plugin. I guess you want me to add this test case in the `ext-plugin-post-resp` plugin.
   However, an empty response body usually occurs when the upstream returns a 304 status code or the request method is HEAD, which needs to be handled separately in subsequent PRs. I have added `TODO`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org