You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/05/06 07:03:23 UTC
[GitHub] [apisix] sandy420 opened a new issue #4160: bug: Seek help !Problems of Prometheus, This should be a bug, please fix it!!!
sandy420 opened a new issue #4160:
URL: https://github.com/apache/apisix/issues/4160
### Issue description
### Environment
* apisix 2.4
* centos 7.6
* nginx version: openresty/1.19.3.1
* etcdctl version: 3.4.13
* apisix-dashboard 2.5
My APISIX 2.4 Prometheus plug-in reported "500 Internal Server Error" after running for a period of time. After restarting apisix, it returned to normal, and a 500 error was reported after a while
1.error.log
`2021/04/29 18:11:58 [warn] 17627#17627: *17251 [lua] ctx.lua:153: __index(): failed to fetch cookie value by key: cookie_ymmoa_passport error: no cookie foun
d in the current request, client: 10.13.65.71, server: , request: "POST /inner/loaner/query-user-auth-info HTTP/1.1", host: "loaner.dev-ag.56qq.com"
2021/04/29 18:11:58 [warn] 17627#17627: *17251 [lua] ctx.lua:153: __index(): failed to fetch cookie value by key: cookie_dev_passport error: no cookie found
in the current request, client: 10.13.65.71, server: , request: "POST /inner/loaner/query-user-auth-info HTTP/1.1", host: "loaner.dev-ag.56qq.com"
2021/04/29 18:12:00 [error] 17627#17627: *18598 lua entry thread aborted: runtime error: ...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua:206: att
empt to compare number with string
stack traceback:
coroutine 0:
...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua: in function 'set_modify_index'
...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua:229: in function 'etcd_modify_index'
...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua:295: in function 'handler'
/home/poseidon/apisix-2.4/apisix/api_router.lua:130: in function 'handler'
...eidon/apisix-2.4//deps/share/lua/5.1/resty/radixtree.lua:720: in function 'dispatch'
/home/poseidon/apisix-2.4/apisix/api_router.lua:165: in function 'match'
/home/poseidon/apisix-2.4/apisix/init.lua:292: in function 'http_access_phase'
access_by_lua(nginx.conf:205):2: in main chunk, client: 10.10.10.100, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "10.10.10.100:80"
2021/04/29 18:12:15 [error] 17627#17627: *19213 lua entry thread aborted: runtime error: ...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua:206: att
empt to compare number with string
stack traceback:
coroutine 0:
...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua: in function 'set_modify_index'
...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua:229: in function 'etcd_modify_index'
...seidon/apisix-2.4/apisix/plugins/prometheus/exporter.lua:295: in function 'handler'
/home/poseidon/apisix-2.4/apisix/api_router.lua:130: in function 'handler'
...eidon/apisix-2.4//deps/share/lua/5.1/resty/radixtree.lua:720: in function 'dispatch'
/home/poseidon/apisix-2.4/apisix/api_router.lua:165: in function 'match'
/home/poseidon/apisix-2.4/apisix/init.lua:292: in function 'http_access_phase'
access_by_lua(nginx.conf:205):2: in main chunk, client: 10.10.10.100, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "10.10.10.100:80"
`
2.my config.yaml:
etcd:
host:
- "http://10.10.0.10:2379"
apisix:
admin_key:
- name: "admin"
# yamllint disable rule:comments-indentation
key: 123456789 # using fixed API token has security risk, please update it when you deploy to production environment
# yamllint enable rule:comments-indentation
role: admin
plugins:
- prometheus
- log-rotate
- proxy-rewrite
- redirect
plugin_attr:
prometheus:
export_uri: /apisix/prometheus/metrics
server-info:
report_interval: 60
report_ttl: 3600
log-rotate:
interval: 86400
max_kept: 30
nginx_config:
http:
access_log_format: "$time_iso8601|$msec|$status|$request_completion|$bytes_sent|$body_bytes_sent|$realip_remote_addr|$remote_addr|$http_x_forwarded_for|$
remote_user|$host|$server_name|$server_port|$server_protocol|$scheme|$request_method|$request_length|$request_time|$request_uri|$uri|$content_length|$content_type|$http_referer|$http_user_agent|$http_app_jb|$http_client_info|$upstream_addr|$upstream_connect_time|$upstream_header_time|$upstream_response_time|$upstream_status|$upstream_bytes_received|$upstream_cache_status|$upstream_http_content_type|$upstream_http_content_length|$upstream_http_content_disposition|$http_x_cat_parent_id|$upstream_scheme://$upstream_host$upstream_uri"
Seek help !How to solve this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-833159411
> > > I can't reproduce your exception, please describe clearly the configuration of config.yaml and the detailed steps you took to cause the exception, it's important for me to reproduce this.
> > > This is my config.yaml
> > > 
>
> The indentation of the server_info plugin configuration is wrong, it should be like this:
>
> ```yaml
> plugin_attr:
> prometheus:
> export_uri: /apisix/prometheus/metrics
> server-info:
> report_interval: 600,
> report_ttl: 3600
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829853499
> > I can't reproduce your exception, please describe clearly the configuration of config.yaml and the detailed steps you took to cause the exception, it's important for me to reproduce this.
> > This is my config.yaml
> > 
The indentation of the server_info plugin configuration is wrong, it should be like this:
```yaml
plugin_attr:
prometheus:
export_uri: /apisix/prometheus/metrics
server-info:
report_interval: 600,
report_ttl: 3600
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #4160: bug: Seek help !Problems of Prometheus, This should be a bug, please fix it!!!
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-833340213
my reproduction process
1. start apisix 2.4
2. add route
```json
{
"update_time": 1619936610,
"uri": "/hello",
"create_time": 1619936610,
"id": "1",
"status": 1,
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
},
"scheme": "http",
"hash_on": "vars",
"pass_host": "pass"
},
"priority": 0
}
```
3. `curl -i http://127.0.0.1:9080/apisix/prometheus/metrics`
`item.modifiedIndex` is a number type value.
what did you do to cause `item.modifiedIndex` to become a string type value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829119331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 closed issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 closed issue #4160:
URL: https://github.com/apache/apisix/issues/4160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829720831
> I can't reproduce your exception, please describe clearly the configuration of config.yaml and the detailed steps you took to cause the exception, it's important for me to reproduce this.
This is my config.yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829121022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 removed a comment on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 removed a comment on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829119331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #4160: bug: Seek help !Problems of Prometheus, This should be a bug, please fix it!!!
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-833468542
@sandy420 good catch! welcome PR!
I can reproduce it now
step 1：create a plugin config
```
curl http://127.0.0.1:9080/apisix/admin/plugin_configs/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
{
"desc": "blah",
"plugins": {
"prometheus":{}
}
}'
```
step2: bind it to route
```
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
{
"uris": ["/hello"],
"plugin_config_id": 1,
"plugins": {
"limit-count": {
"count": 20,
"time_window": 60,
"rejected_code": 503
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
}
}'
```
step3: trigger route
```
curl 127.0.0.1:9080/hello
```
step4: request prometheus endpoint
```
curl http://127.0.0.1:9080/apisix/prometheus/metrics
```
if it does not appear, repeat step4.
the error.log is
```
2021/05/06 20:00:17 [error] 32148#32148: *480 lua entry thread aborted: runtime error: /usr/local/apisix/apisix/plugins/prometheus/exporter.lua:209: attempt to compare number with string
stack traceback:
coroutine 0:
/usr/local/apisix/apisix/plugins/prometheus/exporter.lua: in function 'set_modify_index'
/usr/local/apisix/apisix/plugins/prometheus/exporter.lua:232: in function 'etcd_modify_index'
/usr/local/apisix/apisix/plugins/prometheus/exporter.lua:298: in function 'handler'
/usr/local/apisix/apisix/api_router.lua:130: in function 'handler'
/usr/local/apisix//deps/share/lua/5.1/resty/radixtree.lua:720: in function 'dispatch'
/usr/local/apisix/apisix/api_router.lua:165: in function 'match'
/usr/local/apisix/apisix/init.lua:292: in function 'http_access_phase'
access_by_lua(nginx.conf:204):2: in main chunk, client: 127.0.0.1, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "127.0.0.1:9080"
```
this problem may be caused by https://github.com/apache/apisix/blob/b551f42d90d07598efd775fdeac392043534f8b4/apisix/plugin_config.lua#L65
@spacewander pls take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-833278523
This should be a bug, please fix it!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #4160: bug: Seek help !Problems of Prometheus, This should be a bug, please fix it!!!
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-833350765
This is one of my route configurations:
{
"methods": [
"HEAD",
"POST",
"GET",
"PUT",
"DELETE",
"OPTIONS",
"PATCH"
],
"id": "00000000000000007643",
"hosts": [
"xxxx.xxxx.com"
],
"status": 1,
"desc": "",
"update_time": 1619686104,
"create_time": 1618974733,
"enable_websocket": false,
"plugin_config_id": "666",
"name": "xxxxx",
"upstream_id": "345631436537070303",
"uris": [
"/xxxxx/*"
],
"plugins": {
"proxy-rewrite": {
"disable": false,
"host": "xxxxx.xxxx.com",
"headers": {
"primeval_host": "$host",
"Cookie": "xxx_online=$cookie_xxx_passport;xxx_passport=$cookie_dev_passport; $http_cookie"
}
}
},
"priority": 0
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #4160: bug: Seek help !Problems of Prometheus, This should be a bug, please fix it!!!
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-834002003
@tzssangglass
Thanks for your investigation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers closed issue #4160: bug: Seek help !Problems of Prometheus, This should be a bug, please fix it!!!
Posted by GitBox <gi...@apache.org>.
tokers closed issue #4160:
URL: https://github.com/apache/apisix/issues/4160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829305672
You should provide relevant configuration information for this route to help diagnose the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #4160: bug: Seek help !Problems of Prometheus, a plug-in of APISIX 2.4.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #4160:
URL: https://github.com/apache/apisix/issues/4160#issuecomment-829194196
I can't reproduce your exception, please describe clearly the configuration of config.yaml and the detailed steps you took to cause the exception, it's important for me to reproduce this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org