You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "ranxuxin (via GitHub)" <gi...@apache.org> on 2023/03/06 04:28:52 UTC
[GitHub] [apisix] ranxuxin opened a new issue, #9007: bug: apisix core dump when receiving grpc client stream
ranxuxin opened a new issue, #9007:
URL: https://github.com/apache/apisix/issues/9007
### Current Behavior
There're too many tcp connections when using http1.1 to communicate with etcd cluster before apisix 3.1.0. To reduce
connections, I upgrade apisix to 3.1.0 to use grpc connecting to etcd cluster. At beginning, it does reduce connections to etcd cluster. However, it generates core dump without any manually operation in the early morning.
Apisix's error log indicates that Apisix 3.1.0 generate core dump while receiving grpc client stream. Is the function that grpc connects to etcd cluster in apisix 3.1.0 still experimental? When could it be ready for productive environment, please?
use gdb to show backstack of core dump generated by openresty is as below:
(gdb) bt
warning: skipping .debug_frame info of /usr/local/openresty/libgrpc_engine.so: Found an FDE when not expecting it.
#0 0x00007fa821c631c1 in runtime.raise.abi0 () from /usr/local/openresty/libgrpc_engine.so
#1 0x00007fa821c47365 in runtime.dieFromSignal () from /usr/local/openresty/libgrpc_engine.so
#2 0x000000c000000006 in ?? ()
#3 0x00007fa821c3075a in runtime.preprintpanics () from /usr/local/openresty/libgrpc_engine.so
#4 0x000000c009d23ac8 in ?? ()
#5 0x00007fa821c31d2d in runtime.fatalpanic () from /usr/local/openresty/libgrpc_engine.so
#6 0x000000c000000006 in ?? ()
#7 0x01007fa821c30909 in ?? ()
#8 0x00007fa821c31d60 in ?? () from /usr/local/openresty/libgrpc_engine.so
#9 0x000000c009d23b50 in ?? ()
#10 0x000000c00abd36c0 in ?? ()
#11 0x00007fa821c313fa in runtime.gopanic () from /usr/local/openresty/libgrpc_engine.so
#12 0x000000c009d23ad8 in ?? ()
#13 0x000000c009d23a97 in ?? ()
#14 0x000000c009d23b88 in ?? ()
#15 0x00007fa821c313fa in runtime.gopanic () from /usr/local/openresty/libgrpc_engine.so
#16 0x0000000000000000 in ?? ()
### Expected Behavior
_No response_
### Error Logs
the panic backstack in apisix error log as below:
2023/03/06 05:57:08 [notice] 4157#4157: signal 17 (SIGCHLD) received from 4164
2023/03/06 05:57:08 [alert] 4157#4157: worker process 4164 exited on signal 6 (core dumped)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7fa821f82220]
goroutine 5384582 [running]:
google.golang.org/grpc/internal/status.(*Error).GRPCStatus(0x7fa82200fbe6?)
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/internal/status/status.go:155
google.golang.org/grpc/status.FromError({0x7fa8221db020, 0x0})
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/status/status.go:94 +0xea
google.golang.org/grpc.toRPCErr({0x7fa8221db020, 0x0})
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/rpc_util.go:848 +0x288
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00986adc0, {0x7fa8221c5e40?, 0xc0097c0090}, 0x1?)
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/stream.go:1002 +0x3ba
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc009f9984c?)
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/stream.go:845 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc009f997a0, 0xc00ad6cf20, 0xc00ad6cef0)
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/stream.go:705 +0x2f1
google.golang.org/grpc.(*clientStream).RecvMsg(0xc009f997a0, {0x7fa8221c5e40?, 0xc0097c0090?})
/root/go/pkg/mod/google.golang.org/grpc@v1.47.0/stream.go:844 +0x11f
github.com/api7/grpc-client-nginx-module/conn.(*Stream).Recv(0xc00ad6cfb0?)
/tmp/tmp.tYflEs0Gp1/grpc-client-nginx-module-v0.4.1/grpc-engine/conn/grpc.go:192 +0x7a
main.grpc_engine_stream_recv.func1()
/tmp/tmp.tYflEs0Gp1/grpc-client-nginx-module-v0.4.1/grpc-engine/main.go:233 +0x26
created by main.grpc_engine_stream_recv
/tmp/tmp.tYflEs0Gp1/grpc-client-nginx-module-v0.4.1/grpc-engine/main.go:232 +0x91
### Steps to Reproduce
1. configure apisix3.1.0 as below:
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_key:
- name: admin
key: ilikecocacola # using fixed API token has security risk, please update it when you deploy to production environment
role: admin
etcd:
use_grpc: true
host:
- "http://10.26.21.245:12378"
nginx_config:
error_log: /data0/apisix_error.log
error_log_level: notice
worker_processes: auto
### Environment
apisix 3.1.0
openresty/1.21.4.1
centos 7 linux kernel:3.10.0-862.14.4.el7.x86_64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "spacewander (via GitHub)" <gi...@apache.org>.
spacewander commented on issue #9007:
URL: https://github.com/apache/apisix/issues/9007#issuecomment-1459105233
This part of code is in [grpc/grpc-go](https://github.com/grpc/grpc-go/blob/682ed1a71602fa94a9b9c56d2b2b2a716bccd036/internal/status/status.go#L155). Maybe you can discuss this with them. We (as the user) can't override it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ranxuxin commented on issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "ranxuxin (via GitHub)" <gi...@apache.org>.
ranxuxin commented on issue #9007:
URL: https://github.com/apache/apisix/issues/9007#issuecomment-1484029174
@spacewander I'll find the core dump file. By the way, I met the performance problem while operating a route using admin request. #9157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] monkeyDluffy6017 closed issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 closed issue #9007: bug: apisix core dump when receiving grpc client stream
URL: https://github.com/apache/apisix/issues/9007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ranxuxin commented on issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "ranxuxin (via GitHub)" <gi...@apache.org>.
ranxuxin commented on issue #9007:
URL: https://github.com/apache/apisix/issues/9007#issuecomment-1484468997
你好，泽轩，
那个core dump 只在那一台部署的apisix并且配置的etcd grpc机器上出现过。之后又部署apisix和相同etcd grpc配置到其他几台机器上，都没再复现这个core dump。 应该是系统环境问题。
另外，那台机器重新初始化了，core dump也被删除了。 不过在github 里仍然能看到 core dump 的堆栈。 #9007 . 由于openresty 的编译优化，用 gdb 的 f 1 也看不到代码内容。
祝好，
冉旭欣
At 2023-03-22 10:19:08, "罗泽轩" ***@***.***> wrote:
@ranxuxin
Could you provide the coredump for this?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "spacewander (via GitHub)" <gi...@apache.org>.
spacewander commented on issue #9007:
URL: https://github.com/apache/apisix/issues/9007#issuecomment-1457383072
Look like it crashed in https://github.com/grpc/grpc-go/blob/682ed1a71602fa94a9b9c56d2b2b2a716bccd036/internal/status/status.go#L155, which is called in https://github.com/grpc/grpc-go/blob/682ed1a71602fa94a9b9c56d2b2b2a716bccd036/status/status.go#L94
```
func FromError(err error) (s *Status, ok bool) {
if err == nil {
return nil, true
}
if se, ok := err.(interface {
GRPCStatus() *Status
}); ok {
return se.GRPCStatus(), true
}
return New(codes.Unknown, err.Error()), false
}
```
It's quite strange as this function has checked if `err` is nil before calling it.
Can it be reproduced by running another client writing in Go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ranxuxin commented on issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "ranxuxin (via GitHub)" <gi...@apache.org>.
ranxuxin commented on issue #9007:
URL: https://github.com/apache/apisix/issues/9007#issuecomment-1457781507
@spacewander Thank to your response. The codes above is the backtrace while I met the core dump. The err variable is not nil. And the err is asserted successfully as the Error type which has GRPCStatus() function. But it cannot guarantee that the status field of Error struct is not nil. Maybe not all error generated from the package have the non-nil status.
To fix this defect, could we overide the GRPCStatus() function as below？
func (e *Error) GRPCStatus() *Status {
if e.s == nil {
return New(codes.Unknown, err.Error())
}
return e.s
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #9007: bug: apisix core dump when receiving grpc client stream
Posted by "spacewander (via GitHub)" <gi...@apache.org>.
spacewander commented on issue #9007:
URL: https://github.com/apache/apisix/issues/9007#issuecomment-1478830944
@ranxuxin
Could you provide the coredump for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org