You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2020/09/09 02:00:16 UTC
[GitHub] [apisix] fankeke opened a new issue #2188: request help:
fankeke opened a new issue #2188:
URL: https://github.com/apache/apisix/issues/2188
### Issue description
plugin： url_blocker.lua
```
local from = re_find(ctx.var.request_uri, conf.block_rules_concat, "jo")
```
Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690843717
There is also a library to match plain text in parallel: https://github.com/cloudflare/lua-aho-corasick.
@fankeke
Can we divide the 1+ million block urls into groups according to their hosts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #2188:
URL: https://github.com/apache/apisix/issues/2188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fankeke commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
fankeke commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690832270
Improving performance mainly depends on the quality of the rules.
=======
thanks for reply
In our system, there are already 1+ million block urls， if every request should be match with each url urls， I'm afraid it won't work. :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-773966559
Closed as lack of response.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fankeke commented on issue #2188: request help:
Posted by GitBox <gi...@apache.org>.
fankeke commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-689251456
@membphis
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690842830
> Improving performance mainly depends on the quality of the rules.
>
> =======
> thanks for reply
>
> In our system, there are already 1+ million block urls， if every request should be match with each url urls， I'm afraid it won't work. :(
Then a good data structure (something like Trie Tree, or AC Automata?) might be used to optimize the match process, :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690842830
> Improving performance mainly depends on the quality of the rules.
>
> =======
> thanks for reply
>
> In our system, there are already 1+ million block urls， if every request should be match with each url urls， I'm afraid it won't work. :(
Then a good data structure (something like Trie Tree, or AC Automata?) might be used to optimize the match process, :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690842830
> Improving performance mainly depends on the quality of the rules.
>
> =======
> thanks for reply
>
> In our system, there are already 1+ million block urls， if every request should be match with each url urls， I'm afraid it won't work. :(
Then a good data structure (something like Trie Tree, or AC Automata?) might be used to optimize the match process, :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fankeke commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
fankeke commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-692671931
Can we divide the 1+ million block urls into groups according to their hosts?
===========
@spacewander the whole 1+million urls only for one host(upstream) :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fankeke commented on issue #2188: request help:
Posted by GitBox <gi...@apache.org>.
fankeke commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-689251844
Every request will be handled by block plugin.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-692680602
> Can we divide the 1+ million block urls into groups according to their hosts?
>
> ===========
>
> @spacewander the whole 1+million urls only for one host(upstream) :)
Can you give some examples?
On the other hand, why are there a million regex rules under the same host?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fankeke removed a comment on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
fankeke removed a comment on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-689251844
Every request will be handled by block plugin.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-692679685
> Improving performance mainly depends on the quality of the rules.
>
> For runtime selection, if better performance is required, streaming matching such as hyperscan is needed.
uri match is none business with streaming match
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690843717
There is also a library to match plain text in parallel: https://github.com/cloudflare/lua-aho-corasick.
@fankeke
Can we divide the 1+ million block urls into groups according to their hosts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690034015
you can take a look at this benchmark result: https://rust-leipzig.github.io/regex/2017/03/28/comparison-of-regex-engines/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690032838
Improving performance mainly depends on the quality of the rules.
For runtime selection, if better performance is required, streaming matching such as hyperscan is needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690843717
There is also a library to match plain text in parallel: https://github.com/cloudflare/lua-aho-corasick.
@fankeke
Can we divide the 1+ million block urls into groups according to their hosts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fankeke commented on issue #2188: request help: url_block.lua plugin: Is there any performance bottleneck if there are many block rules, say 1000000 url rules in production enviroment
Posted by GitBox <gi...@apache.org>.
fankeke commented on issue #2188:
URL: https://github.com/apache/apisix/issues/2188#issuecomment-690832270
Improving performance mainly depends on the quality of the rules.
=======
thanks for reply
In our system, there are already 1+ million block urls， if every request should be match with each url urls， I'm afraid it won't work. :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org