You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/09/02 02:07:37 UTC
[GitHub] [apisix] okaybase opened a new pull request #4965: feat(proxy-mirror): support mirror requests quota
okaybase opened a new pull request #4965:
URL: https://github.com/apache/apisix/pull/4965
### What this PR does / why we need it:
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
support mirror requests quota
### Pre-submission checklist:
* [x] Did you explain what problem does this PR solve? Or what new features have been added?
* [x] Have you added corresponding test cases?
* [x] Have you modified the corresponding document?
* [x] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix/tree/master#community) first**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703456004
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -53,7 +59,18 @@ function _M.rewrite(conf, ctx)
core.log.info("proxy mirror plugin rewrite phase, conf: ", core.json.delay_encode(conf))
ctx.var.upstream_host = ctx.var.host
- ctx.var.upstream_mirror_host = conf.host
+
+ if not conf.sample_ratio or conf.sample_ratio == 1 then
+ ctx.var.upstream_mirror_host = conf.host
+ else
+ local val = math_random()
+ core.log.info("mirror request sample_ratio conf: ", conf.sample_ratio,
+ ", random value: ", val)
Review comment:
okay~ done~ thanks a lot~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on pull request #4965: feat(proxy-mirror): support mirror percentage
Posted by GitBox <gi...@apache.org>.
okaybase commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911756467
Will use random sampling percentages to achieve this function~ @spacewander @tokers @tzssangglass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701641380
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
Logs are currently checked~
50% it might not be very very accurate in the random case~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #4965:
URL: https://github.com/apache/apisix/pull/4965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703259350
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
Could give some tips~ @spacewander
Current the hit count is checked via: `qr/(uri: \/hello\?sample_ratio=0\.5){75,125}/`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703259350
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
Could you give some tips~ @spacewander
Current the hit count is checked via: `qr/(uri: \/hello\?sample_ratio=0\.5){75,125}/`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r705128387
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
okay~ will write a new backend API~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701637199
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -26,6 +26,12 @@ local schema = {
pattern = [[^http(s)?:\/\/[a-zA-Z0-9][-a-zA-Z0-9]{0,62}]]
.. [[(\.[a-zA-Z0-9][-a-zA-Z0-9]{0,62})+(:[0-9]{1,5})?$]],
},
+ percentage = {
Review comment:
okay~ done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-917006155
According to the error.log, it seems that the mirror request may not
finish when the HTTP request is done, if the CPU resource is low. So just roll back to check it in the error log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r704976733
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +455,163 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local http = require("resty.http")
+ local core = require("apisix.core")
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ -- reset count
+ local count = 0
+ local httpc = http.new()
+ local url = "http://127.0.0.1:1980/stat_count?action=reset"
+ local res, err = httpc:request_uri(url, {method = "GET"})
+ if not res then
+ core.log.error(err)
+ else
+ core.log.info("reset the mirror request stat count to " .. res.body)
+ count = res.body
+ end
+
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
+ end
+
+ url = "http://127.0.0.1:1980/stat_count"
+ res, err = httpc:request_uri(url, {method = "GET"})
+ if not res then
+ core.log.error(err)
+ else
+ core.log.info("the mirror request stat count is " .. res.body)
+ count = res.body
+ end
+ assert(count >= 75 and count <= 125)
+ elseif code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- error_log_like eval
+qr/(uri: \/hello\?sample_ratio=0\.5){75,125}/
+--- timeout: 60
Review comment:
We can send requests parallelly like https://github.com/apache/apisix/blob/master/docs/en/latest/internal/testing-framework.md#send-request
Remember to use a high worker_connections in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase edited a comment on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
okaybase edited a comment on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911069673
> Could you give some use cases about this feature?
okay~
In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701530806
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -53,7 +60,18 @@ function _M.rewrite(conf, ctx)
core.log.info("proxy mirror plugin rewrite phase, conf: ", core.json.delay_encode(conf))
ctx.var.upstream_host = ctx.var.host
- ctx.var.upstream_mirror_host = conf.host
+
+ if not conf.percentage or conf.percentage == 1 then
+ ctx.var.upstream_mirror_host = conf.host
+ else
+ math.randomseed(socket.gettime())
Review comment:
thanks~ done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r704077672
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
Thanks firstly~
According to the document, seems the mirror address only support host~
@spacewander
https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/proxy-mirror.md#attributes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701706770
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
TO design the almost range, I wrote a python script:
```
#!/usr/bin/env python3
# coding: utf-8
curr = [1, 1]
turns = 180
for i in range(1, turns+1):
prev = curr
curr = []
curr.append(1)
for j in range(1, i):
curr.append(prev[j-1] + prev[j])
curr.append(1)
n = sum(curr)
for i in range(turns+1):
print("%d / %d is %f" % (i, turns, curr[i] / float(n)))
exp_per = 0.999
mid = int(turns / 2)
for i in range(1, mid):
n_per = 0
for j in range(mid - i, mid + i + 1):
n_per += curr[j] / float(n)
if n_per > exp_per:
print(mid - i, j)
break
if n_per > exp_per:
break
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase edited a comment on pull request #4965: feat(proxy-mirror): support mirror percentage
Posted by GitBox <gi...@apache.org>.
okaybase edited a comment on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911128702
> > > Could you give some use cases about this feature?
> > > okay~
> > > In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
>
> Personally, I prefer to sample randomly than sample the first N. As,
>
> 1. sample randomly is more accurate. It doesn't miss requests after the first N
> 2. we don't need a place to store the count.
>
> What about your opinion?
> @membphis @tokers @tzssangglass
Thanks for your careful review and suggestions firstly~ :smile:
Use random sampling percentages to achieve this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701706770
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
TO design the almost range, I wrote a python script:
```
#!/usr/bin/env python3
# coding: utf-8
curr = [1, 1]
turns = 180
for i in range(1, turns+1):
prev = curr
curr = []
curr.append(1)
for j in range(1, i):
curr.append(prev[j-1] + prev[j])
curr.append(1)
n = sum(curr)
for i in range(turns+1):
print("%d / %d is %f" % (i, turns, curr[i] / float(n)))
exp_per = 0.999
mid = int(turns / 2)
for i in range(1, mid):
n_per = 0
for j in range(mid - i, mid + i + 1):
n_per += curr[j] / float(n)
if n_per > exp_per:
print(mid - i, j)
break
if n_per > exp_per:
break
```
So we can expect 99.9% of the result hit from 68 to 112.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701630725
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -26,6 +26,12 @@ local schema = {
pattern = [[^http(s)?:\/\/[a-zA-Z0-9][-a-zA-Z0-9]{0,62}]]
.. [[(\.[a-zA-Z0-9][-a-zA-Z0-9]{0,62})+(:[0-9]{1,5})?$]],
},
+ percentage = {
Review comment:
IMHO, we use sample_ratio for this purpose in skywalking & zipkin. What about using this name?
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
Could we check if almost 50% requests reach the mirror?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703110158
##########
File path: docs/en/latest/plugins/proxy-mirror.md
##########
@@ -29,13 +29,14 @@ The proxy-mirror plugin, which provides the ability to mirror client requests.
| Name | Type | Requirement | Default | Valid | Description |
| ---- | ------ | ----------- | ------- | ----- | --------------------------------------------------------------------------------------------------------------------------- |
-| host | string | optional | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain schema: http or https, not URI part) |
+| host | string | required | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain schema: http or https, not URI part) |
Review comment:
```suggestion
| host | string | required | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain scheme: http or https, and without the path part) |
```
##########
File path: docs/en/latest/plugins/proxy-mirror.md
##########
@@ -29,13 +29,14 @@ The proxy-mirror plugin, which provides the ability to mirror client requests.
| Name | Type | Requirement | Default | Valid | Description |
| ---- | ------ | ----------- | ------- | ----- | --------------------------------------------------------------------------------------------------------------------------- |
-| host | string | optional | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain schema: http or https, not URI part) |
+| host | string | required | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain schema: http or https, not URI part) |
+| sample_ratio | number | optional | 1 | [0.00001, 1] | the sample_ratio of mirror requests |
Review comment:
```suggestion
| sample_ratio | number | optional | 1 | [0.00001, 1] | the sample ratio that requests will be mirrored. |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911107859
>
>
> > Could you give some use cases about this feature?
> > okay~
> > In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
Personally, I prefer to sample randomly than sample the first N. As,
1. sample randomly is more accurate. It doesn't miss requests after the first N
2. we don't need a place to store the count.
What about your opinion?
@membphis @tokers @tzssangglass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703381800
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -53,7 +59,18 @@ function _M.rewrite(conf, ctx)
core.log.info("proxy mirror plugin rewrite phase, conf: ", core.json.delay_encode(conf))
ctx.var.upstream_host = ctx.var.host
- ctx.var.upstream_mirror_host = conf.host
+
+ if not conf.sample_ratio or conf.sample_ratio == 1 then
+ ctx.var.upstream_mirror_host = conf.host
+ else
+ local val = math_random()
+ core.log.info("mirror request sample_ratio conf: ", conf.sample_ratio,
+ ", random value: ", val)
Review comment:
bad indentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r705127530
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +455,163 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local http = require("resty.http")
+ local core = require("apisix.core")
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ -- reset count
+ local count = 0
+ local httpc = http.new()
+ local url = "http://127.0.0.1:1980/stat_count?action=reset"
+ local res, err = httpc:request_uri(url, {method = "GET"})
+ if not res then
+ core.log.error(err)
+ else
+ core.log.info("reset the mirror request stat count to " .. res.body)
+ count = res.body
+ end
+
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
+ end
+
+ url = "http://127.0.0.1:1980/stat_count"
+ res, err = httpc:request_uri(url, {method = "GET"})
+ if not res then
+ core.log.error(err)
+ else
+ core.log.info("the mirror request stat count is " .. res.body)
+ count = res.body
+ end
+ assert(count >= 75 and count <= 125)
+ elseif code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- error_log_like eval
+qr/(uri: \/hello\?sample_ratio=0\.5){75,125}/
+--- timeout: 60
Review comment:
okay, the test case has changed to send requests parallelly, but the ci log don't show more info about the test status 500, could you have a look at it~ @spacewander
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase edited a comment on pull request #4965: feat(proxy-mirror): support mirror percentage
Posted by GitBox <gi...@apache.org>.
okaybase edited a comment on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911128702
> > > Could you give some use cases about this feature?
> > > okay~
> > > In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
>
> Personally, I prefer to sample randomly than sample the first N. As,
>
> 1. sample randomly is more accurate. It doesn't miss requests after the first N
> 2. we don't need a place to store the count.
>
> What about your opinion?
> @membphis @tokers @tzssangglass
Thanks for your careful review and suggestions firstly~ :smile:
Will use random sampling percentages to achieve this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-917014813
> According to the error.log, it seems that the mirror request may not
> finish when the HTTP request is done, if the CPU resource is low. So just roll back to check it in the error log
okay~ thanks a lot~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-913914771
the random sample and test cases done~ @spacewander @tokers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r702023618
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
okay~ thanks a lot~ done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703966617
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
We can add a `/inc` API, it will return an increasing number each call. Like,
```
GET /inc
1
GET /inc
2
GET /inc
3
```
So by hitting the API we can know how many requests are mirrored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
tokers commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911475113
> > > Could you give some use cases about this feature?
> > > okay~
> > > In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
>
> Personally, I prefer to sample randomly than sample the first N. As,
>
> 1. sample randomly is more accurate. It doesn't miss requests after the first N
> 2. we don't need a place to store the count.
>
> What about your opinion?
> @membphis @tokers @tzssangglass
I think a random sample is better than the current way. Also, if we support these rate-limiting ways, we have to change multiple plugins in the future if new rate-limiting algorithms are introduced, and it's prone to be inconsistent on the configurations items.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r702413911
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
Could we write a new backend API to https://github.com/apache/apisix/blob/0e2b284d8086fefdf7cde5a658886323f98ac467/t/lib/server.lua#L428 and count the hit number?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911189153
randoml +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r704077672
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
Thanks firstly~ @spacewander
According to the document, seems the mirror address only support host~
https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/proxy-mirror.md#attributes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r702023618
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
okay~ thanks a lot~ done~ @spacewander
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass edited a comment on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
tzssangglass edited a comment on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911189153
random +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r703116437
##########
File path: docs/en/latest/plugins/proxy-mirror.md
##########
@@ -29,13 +29,14 @@ The proxy-mirror plugin, which provides the ability to mirror client requests.
| Name | Type | Requirement | Default | Valid | Description |
| ---- | ------ | ----------- | ------- | ----- | --------------------------------------------------------------------------------------------------------------------------- |
-| host | string | optional | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain schema: http or https, not URI part) |
+| host | string | required | | | Specify a mirror service address, e.g. http://127.0.0.1:9797 (address needs to contain schema: http or https, not URI part) |
+| sample_ratio | number | optional | 1 | [0.00001, 1] | the sample_ratio of mirror requests |
Review comment:
thanks~ done~ @tokers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701520436
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -53,7 +60,18 @@ function _M.rewrite(conf, ctx)
core.log.info("proxy mirror plugin rewrite phase, conf: ", core.json.delay_encode(conf))
ctx.var.upstream_host = ctx.var.host
- ctx.var.upstream_mirror_host = conf.host
+
+ if not conf.percentage or conf.percentage == 1 then
+ ctx.var.upstream_mirror_host = conf.host
+ else
+ math.randomseed(socket.gettime())
Review comment:
`randomseed` was already called in `http_init_worker`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
okaybase commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911069673
> Could you give some use cases about this feature?
In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-916772884
Try this patch?
```
diff --git t/plugin/proxy-mirror.t t/plugin/proxy-mirror.t
index 888e9a86..a6b7e3ca 100644
--- t/plugin/proxy-mirror.t
+++ t/plugin/proxy-mirror.t
@@ -38,7 +38,7 @@ add_block_preprocessor(sub {
local http = require("resty.http")
local httpc = http.new()
- local url = "http://127.0.0.1:1984/stat_count?action=inc"
+ local url = "http://127.0.0.1:1980/stat_count?action=inc"
local res, err = httpc:request_uri(url, {method = "GET"})
if not res then
core.log.error(err)
@@ -588,46 +588,14 @@ passed
-=== TEST 17: set route(stat_count)
---- config
- location /t {
- content_by_lua_block {
- local t = require("lib.test_admin").test
- local code, body = t('/apisix/admin/routes/2',
- ngx.HTTP_PUT,
- [[{
- "upstream": {
- "nodes": {
- "127.0.0.1:1980": 1
- },
- "type": "roundrobin"
- },
- "uri": "/stat_count"
- }]]
- )
-
- if code >= 300 then
- ngx.status = code
- end
- ngx.say(body)
- }
- }
---- request
-GET /t
---- error_code: 200
---- response_body
-passed
-
-
-
-=== TEST 18: send batch requests and get mirror stat count
+=== TEST 17: send batch requests and get mirror stat count
--- config
location /t {
content_by_lua_block {
local t = require("lib.test_admin").test
-- reset count
- local code, count = t('/stat_count?action=reset', ngx.HTTP_GET)
+ local code, _, count = t('/stat_count?action=reset', ngx.HTTP_GET)
if code >= 300 then
ngx.status = code
end
@@ -645,10 +613,12 @@ passed
end
-- get mirror stat count
- code, count = t('/stat_count', ngx.HTTP_GET)
+ local code, _, count = t('/stat_count', ngx.HTTP_GET)
if code >= 300 then
ngx.status = code
+ return
end
+ count = tonumber(count)
assert(count >= 75 and count <= 125, "mirror request count not in [75, 125]")
}
@@ -657,4 +627,3 @@ passed
GET /t
--- error_log_like eval
qr/(uri: \/hello\?sample_ratio=0\.5){75,125}/
---- timeout: 60
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701655958
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
What about trying it 180 times? Just like https://github.com/apache/apisix/blob/master/docs/en/latest/internal/testing-framework.md#send-request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701641380
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,143 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid percentage)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"percentage\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests percentage to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with percentage 1
+--- request
+GET /hello?percentage=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?percentage=1/
+
+
+
+=== TEST 16: set mirror requests percentage to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "percentage": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 17: hit route with percentage 0.5
Review comment:
50% it might not be very very accurate in the random case~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r704348533
##########
File path: t/plugin/proxy-mirror.t
##########
@@ -445,3 +445,139 @@ GET /t
passed
--- no_error_log
[error]
+
+
+
+=== TEST 13: sanity check (invalid sample_ratio)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 10
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.print(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 400
+--- response_body
+{"error_msg":"failed to check the configuration of plugin proxy-mirror err: property \"sample_ratio\" validation failed: expected 10 to be smaller than 1"}
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: set mirror requests sample_ratio to 1
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- error_code: 200
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: hit route with sample_ratio 1
+--- request
+GET /hello?sample_ratio=1
+--- error_code: 200
+--- response_body
+hello world
+--- error_log_like eval
+qr/uri: \/hello\?sample_ratio=1/
+
+
+
+=== TEST 16: set mirror requests sample_ratio to 0.5
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "proxy-mirror": {
+ "host": "http://127.0.0.1:1986",
+ "sample_ratio": 0.5
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]]
+ )
+
+ if code == 200 then
+ for i = 1, 200 do
+ t('/hello?sample_ratio=0.5', ngx.HTTP_GET)
Review comment:
Yes, both the client and mirror hit the /inc API, so the number of request from mirror is total - client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on pull request #4965: feat(proxy-mirror): support mirror requests quota
Posted by GitBox <gi...@apache.org>.
okaybase commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-911128702
> > > Could you give some use cases about this feature?
> > > okay~
> > > In the simulation environment, we hope to use a small amount of real traffic to verify the new function, but the real traffic may be huge, which puts great pressure on the simulation service.
>
> Personally, I prefer to sample randomly than sample the first N. As,
>
> 1. sample randomly is more accurate. It doesn't miss requests after the first N
> 2. we don't need a place to store the count.
>
> What about your opinion?
> @membphis @tokers @tzssangglass
Thanks for your careful review and suggestions firstly~ :smile:
Perhaps random sampling could be one of the policies(random/local/redis/redis-cluster).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701547185
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -15,7 +15,7 @@
-- limitations under the License.
--
local core = require("apisix.core")
-
+local math = math
Review comment:
```suggestion
local math_random = math.random
```
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -53,7 +59,17 @@ function _M.rewrite(conf, ctx)
core.log.info("proxy mirror plugin rewrite phase, conf: ", core.json.delay_encode(conf))
ctx.var.upstream_host = ctx.var.host
- ctx.var.upstream_mirror_host = conf.host
+
+ if not conf.percentage or conf.percentage == 1 then
+ ctx.var.upstream_mirror_host = conf.host
+ else
+ local val = math.random()
Review comment:
```suggestion
local val = math_random()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #4965: feat(proxy-mirror): support mirror requests sample_ratio
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#issuecomment-916775769
Since the patch is so big, I decide to push it to this branch instead (also trigger the CI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] okaybase commented on a change in pull request #4965: feat(proxy-mirror): support mirror requests percentage
Posted by GitBox <gi...@apache.org>.
okaybase commented on a change in pull request #4965:
URL: https://github.com/apache/apisix/pull/4965#discussion_r701637199
##########
File path: apisix/plugins/proxy-mirror.lua
##########
@@ -26,6 +26,12 @@ local schema = {
pattern = [[^http(s)?:\/\/[a-zA-Z0-9][-a-zA-Z0-9]{0,62}]]
.. [[(\.[a-zA-Z0-9][-a-zA-Z0-9]{0,62})+(:[0-9]{1,5})?$]],
},
+ percentage = {
Review comment:
okay~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org