You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2020/08/02 14:27:55 UTC
[GitHub] [apisix] DHB-liuhong opened a new issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
DHB-liuhong opened a new issue #1970:
URL: https://github.com/apache/apisix/issues/1970
### Issue description
### Environment
* apisix version (cmd: `apisix version`): 1.4.1
* OS: centos 7.6
### Minimal test code / Steps to reproduce the issue
1. set ssl configure by curl -i http://192.168.3.36:9080/apisix/admin/ssl -d 'xxxxxxx'
2. Send a large number of requests to Apisix
### What's the actual result? (including assertion message & call stack if applicable)
Many requests can't find upstrean, return 404, after configuring SSL
error.log:
### What's the expected result?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong edited a comment on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong edited a comment on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-669175575
1. route:
curl http://127.0.0.1:9080/apisix/admin/routes/1 -d '
{
"uri": "/*",
"methods": ["PUT", "GET", "POST", "DELETE", "PATCH", "HEAD", "OPTIONS", "CONNECT", "TRACE"],
"plugins": {
"prometheus":{}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"192.168.3.68:8080": 10,
"192.168.3.68:8081": 10,
"192.168.3.68:8082": 10
}
}
}'
2. set ssl
curl -i http://127.0.0.1:9080/apisix/admin/ssl -d '
{
"cert": "-----BEGIN CERTIFICATE-----
MIIGxTGA1UEBwwG5YyX5LqsMUIwQAYDVQQKDDnk
uK3lm73np7vliqjpgJrkv6Hpm4blm6LmnInpmZDlhazlj7jmlL/kvIHlrqLmiLfl
iIblhazlj7gxJDAiBgNVBAMMGyouZW9zLWJlaWppbmctMS5jbWVjbG91ZC5jbjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJUTJIrt/2BzbrwmSii8zR3y
I8Gd5i5X47Zx46S1W0C8Lfwze10tYgcjkdFBPknubRcRi...............",
"key": "-----BEGIN RSA PRIVATE KEY-----
oXE19iZ2lYmxZauLuKsXh+qIUgoDOItGAxuhomagz6NqF4KNBeKl38HJra1/H3Hf
VFUyBc1xEPGRyDzbSCcJ9YsPv/KpCjH3/KplRr4t3HqVlu71Cf7ZcCYT9O6wbTBc
LW8tRtiAkdZ1r4LxrU/X+cgEzM6Ga+EfZyu3iIBaHi/eDR4AvuhCOSMAelE8PK7V
-----END RSA PRIVATE KEY-----",
"sni": "suzhou.cn"
}'
this ssl key, use openssl to creat. didn't list the complete key
3. seq 1 1000 | xargs -I{} -P 20 curl -i https://suzhou.cn
or seq 1 1000 | xargs -I{} -P 20 s3cmd put file1 s3://test/file{}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-669175575
1. route:
curl http://127.0.0.1:9080/apisix/admin/routes/1 -d '
{
"uri": "/*",
"methods": ["PUT", "GET", "POST", "DELETE", "PATCH", "HEAD", "OPTIONS", "CONNECT", "TRACE"],
"plugins": {
"prometheus":{}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"192.168.3.68:8080": 10,
"192.168.3.68:8081": 10,
"192.168.3.68:8082": 10
}
}
}'
2. set ssl
curl -i http://127.0.0.1:9080/apisix/admin/ssl -d '
{
"cert": "-----BEGIN CERTIFICATE-----
MIIGxTGA1UEBwwG5YyX5LqsMUIwQAYDVQQKDDnk
uK3lm73np7vliqjpgJrkv6Hpm4blm6LmnInpmZDlhazlj7jmlL/kvIHlrqLmiLfl
iIblhazlj7gxJDAiBgNVBAMMGyouZW9zLWJlaWppbmctMS5jbWVjbG91ZC5jbjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJUTJIrt/2BzbrwmSii8zR3y
I8Gd5i5X47Zx46S1W0C8Lfwze10tYgcjkdFBPknubRcRi...............",
"key": "-----BEGIN RSA PRIVATE KEY-----
oXE19iZ2lYmxZauLuKsXh+qIUgoDOItGAxuhomagz6NqF4KNBeKl38HJra1/H3Hf
VFUyBc1xEPGRyDzbSCcJ9YsPv/KpCjH3/KplRr4t3HqVlu71Cf7ZcCYT9O6wbTBc
LW8tRtiAkdZ1r4LxrU/X+cgEzM6Ga+EfZyu3iIBaHi/eDR4AvuhCOSMAelE8PK7V
-----END RSA PRIVATE KEY-----",
"sni": "suzhou.cn"
}'
this ssl key, use openssl to creat. didn't list the complete key
3. seq 1 1000 | xargs -I{} -P 20 curl -i https://suzhou.cn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-669125375
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong closed issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong closed issue #1970:
URL: https://github.com/apache/apisix/issues/1970
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-667809991
1,no etcd cluster, just etcd rpm install.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-668319304
step1: don't set ssl configure, put many objects. one request uploads one object. it's good ,All requests are forwarded and processed normally.
step2: set one ssl configure, put many objects. one request uploads one object. Most of the requests are not forwarded, return 404
step3. Testing v1.2, step1 and step2 are good.
thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-667752098
do you enable the etcd cluster? and you can make a try with APISIX 1.5 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong edited a comment on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong edited a comment on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-669175575
1. route:
```
curl http://127.0.0.1:9080/apisix/admin/routes/1 -d '
{
"uri": "/*",
"methods": ["PUT", "GET", "POST", "DELETE", "PATCH", "HEAD", "OPTIONS", "CONNECT", "TRACE"],
"plugins": {
"prometheus":{}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"192.168.3.68:8080": 10,
"192.168.3.68:8081": 10,
"192.168.3.68:8082": 10
}
}
}'
```
2. set ssl
```
curl -i http://127.0.0.1:9080/apisix/admin/ssl -d '
{
"cert": "-----BEGIN CERTIFICATE-----
MIIGxTGA1UEBwwG5YyX5LqsMUIwQAYDVQQKDDnk
uK3lm73np7vliqjpgJrkv6Hpm4blm6LmnInpmZDlhazlj7jmlL/kvIHlrqLmiLfl
iIblhazlj7gxJDAiBgNVBAMMGyouZW9zLWJlaWppbmctMS5jbWVjbG91ZC5jbjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJUTJIrt/2BzbrwmSii8zR3y
I8Gd5i5X47Zx46S1W0C8Lfwze10tYgcjkdFBPknubRcRi...............",
"key": "-----BEGIN RSA PRIVATE KEY-----
oXE19iZ2lYmxZauLuKsXh+qIUgoDOItGAxuhomagz6NqF4KNBeKl38HJra1/H3Hf
VFUyBc1xEPGRyDzbSCcJ9YsPv/KpCjH3/KplRr4t3HqVlu71Cf7ZcCYT9O6wbTBc
LW8tRtiAkdZ1r4LxrU/X+cgEzM6Ga+EfZyu3iIBaHi/eDR4AvuhCOSMAelE8PK7V
-----END RSA PRIVATE KEY-----",
"sni": "suzhou.cn"
}'
```
this ssl key, use openssl to creat. didn't list the complete key
3. seq 1 1000 | xargs -I{} -P 20 curl -i https://suzhou.cn
or seq 1 1000 | xargs -I{} -P 20 s3cmd put file1 s3://test/file{}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] DHB-liuhong commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
DHB-liuhong commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-670813101
@membphis thank you very much, I redeployed the environment, and I can't reproduce this problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-667867533
Can you provide a small case for this problem? I have no easy way to reproduce this problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-669126534
that is my step:
apisix version: `1.4.1`
1. I set an SSL object and the SNI is `test.com`.
2. add one route
```
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/hello",
"plugins": {
},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
}
}'
```
3. use wrk to call a benchmark, it works fine with no error.
```
$ wrk -t 3 -c 60 "https://test.com:9443/hello"
Running 10s test @ https://test.com:9443/hello
3 threads and 60 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.60ms 1.61ms 124.64ms 97.90%
Req/Sec 4.31k 427.86 6.33k 87.21%
127936 requests in 10.03s, 25.37MB read
Requests/sec: 12759.42
Transfer/sec: 2.53MB
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-669245731
@DHB-liuhong I try to run your case, it works fine at my local machine.
here is the full log: https://gist.github.com/membphis/a0a0251dc87ee99eacac2cc736ee30fd
if you still have a problem, I think I need to check your local machine. if you need this help, please contact us.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] wfgydbu commented on issue #1970: bug: apisix-1.4.1 With SSL configured, numerous tests will fail to match any routes
Posted by GitBox <gi...@apache.org>.
wfgydbu commented on issue #1970:
URL: https://github.com/apache/apisix/issues/1970#issuecomment-717059125
I actually reproduced this issue **on the same machine**.
Details steps are:
1. Setup etcd and apisix 1.5
2. Setup ssl configuration. Then there are two scenarios:
Scenario one:
Testing ordinary http or https requests.
Run`seq 1 1000 | xargs -I{} -P 20 curl -i http://127.0.0.1:9080`, work as expected.
Run`seq 1 1000 | xargs -I{} -P 30 curl -i https://test.cn:9443 -k`, where `test.cn` is the SNI set in admin/ssl, and 9443 is the default ssl port. work as expected as well.
**Next.**
Scenario two:
Testing `s3cmd` with http or https.
`s3cmd` is the Command Line S3 Client and Backup tool for Linux to connect the Amazon S3 data storage service. We use apisix as the gateway in this scenario.
Run
`s3cmd --access_key=test --secret_key=test --host=10.10.10.40:9080 --host-bucket="10.10.10.40:9080/%(bucket)" --no-ssl --signature-v2 put file_1kb s3://bucket`
or `s3cmd --access_key=test --secret_key=test --host="test.cn:443" --host-bucket="test.cn:443/%(bucket)" --ssl --signature-v2 put file_1kb s3://bucket --no-check-certificate`
respectively. Both cases will cause errors in the client side:
```
HTTP/1.1 404 Not Found
Date: Tue, 27 Oct 2020 07:16:48 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX web server
{"error_msg":"failed to match any routes"}
```
About 1/10 failure rate when testing with 20 concurrent requests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org