You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "youngxhui (via GitHub)" <gi...@apache.org> on 2025/06/28 17:25:02 UTC
[PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
youngxhui opened a new pull request, #226:
URL: https://github.com/apache/skywalking-go/pull/226
The online service experienced a panic after integrating SkyWalking.
SkyWalking’s panic should not affect the normal operation of the service.
Each goroutine should be guarded with a recover function.
Currently, many goroutines in the codebase are missing recover protection.
panic log
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x48333b]
goroutine 23 [running]:
google.golang.org/protobuf/encoding/protowire.AppendString(...)
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/encoding/protowire/wire.go:455
google.golang.org/protobuf/internal/impl.appendStringNoZeroValidateUTF8({0xc00338a000?, 0x10a?, 0x1e7?}, {0xc00026a978?}, 0x42d52b?, {0x2?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/codec_gen.go:5113 +0x95
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc000165708, {0xc00338a000?, 0xa90705?, 0x1070293?}, {0xaa53ac?}, {0x8?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/encode.go:139 +0x137
google.golang.org/protobuf/internal/impl.appendMessageSliceInfo({0xc00338a000?, 0x0?, 0x3a00?}, {0x2964e20?}, 0xc0000c5de0, {0x20?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/codec_field.go:485 +0xee
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc000022d18, {0xc00338a000?, 0xa90705?, 0x1070322?}, {0x1072000?}, {0x0?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/encode.go:139 +0x137
google.golang.org/protobuf/internal/impl.appendMessageSliceInfo({0xc00338a000?, 0xc00026aa60?, 0x419d45?}, {0xc000072808?}, 0xc0000b8118, {0x0?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/codec_field.go:485 +0xee
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc000022a88, {0xc00338a000?, 0xc000072808?, 0xc00338a000?}, {0xc00026aaf0?}, {0xe5?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/encode.go:139 +0x137
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc001ed2a01?, {{}, {0x1b4baf0, 0xc001081a70}, {0xc00338a000, 0x0, 0x10703c9}, 0x2})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/encode.go:107 +0x85
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xac?, 0x26?}, {0x0, 0x0, 0x0}, {0x1b4baf0, 0xc001081a70})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/proto/encode.go:166 +0x24e
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0xe0?, 0xd4?, 0x80?}, {0x0, 0x0, 0x0}, {0x1b2da60?, 0xc001081a70?})
/root/go/pkg/mod/google.golang.org/protobuf@v1.33.0/proto/encode.go:125 +0x73
github.com/golang/protobuf/proto.marshalAppend({0x0, 0x0, 0x0}, {0x7f693418b118?, 0xc001081a70?}, 0x0)
/root/go/pkg/mod/github.com/golang/protobuf@v1.5.3/proto/wire.go:40 +0x92
github.com/golang/protobuf/proto.Marshal(...)
/root/go/pkg/mod/github.com/golang/protobuf@v1.5.3/proto/wire.go:23
google.golang.org/grpc/encoding/proto.codec.Marshal({}, {0x180d4e0?, 0xc001081a70?})
/root/go/pkg/mod/google.golang.org/grpc@v1.57.0/encoding/proto/proto.go:45 +0x52
google.golang.org/grpc.encode({0x7f69342c6178?, 0x297be40?}, {0x180d4e0?, 0xc001081a70?})
/root/go/pkg/mod/google.golang.org/grpc@v1.57.0/rpc_util.go:633 +0x3e
google.golang.org/grpc.prepareMsg({0x180d4e0?, 0xc001081a70?}, {0x7f69342c6178?, 0x297be40?}, {0x0, 0x0}, {0x0, 0x0})
/root/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1766 +0xc7
google.golang.org/grpc.(*clientStream).SendMsg(0xc000170000, {0x180d4e0, 0xc001081a70})
/root/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:882 +0xd7
skywalking.apache.org/repo/goapi/collect/language/agent/v3.(*traceSegmentReportServiceCollectClient).Send(0xc00004cad0?, 0xc000e70e10?)
/root/go/pkg/mod/skywalking.apache.org/repo/goapi@v0.0.0-20230314034821-0c5a44bb767a/collect/language/agent/v3/Tracing_grpc.pb.go:61 +0x25
github.com/apache/skywalking-go/agent/reporter.(*gRPCReporter).initSendPipeline.func1()
grpc.go:322 +0x1d1
created by github.com/apache/skywalking-go/agent/reporter.(*gRPCReporter).initSendPipeline in goroutine 1
grpc.go:304 +0x5f
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #226:
URL: https://github.com/apache/skywalking-go/pull/226#issuecomment-3016172465
Could you explain why panic in this case? If there is any code bug, we should fic them rather than recovering.
Because it could cause CPU overload, and happen again, which makes your app overloaded too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "youngxhui (via GitHub)" <gi...@apache.org>.
youngxhui commented on PR #226:
URL: https://github.com/apache/skywalking-go/pull/226#issuecomment-3016363695
Currently, I'm unable to reproduce the issue with a demo. Since we cannot guarantee the safety of goroutines, every goroutine must have a recover .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on PR #226:
URL: https://github.com/apache/skywalking-go/pull/226#issuecomment-3016405829
Please update the CHANGES.md file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "youngxhui (via GitHub)" <gi...@apache.org>.
youngxhui commented on PR #226:
URL: https://github.com/apache/skywalking-go/pull/226#issuecomment-3016251660
I haven’t identified the root cause yet. Many services in our production environment are integrated with SkyWalking, but no similar issues have been observed in other services. Just before the service panic, a large number of error logs were printed:
```
2025-06-29T00:00:36.914+0800 ERROR tracer.go:191 reach max tracing send buffer {"SW_CTx": "[sever-name,instance-name,N/A,N/A,-1]"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #226:
URL: https://github.com/apache/skywalking-go/pull/226#issuecomment-3016298750
We should merge this directly. Instead, we should try to analyze why.
Because, if it is a predicted case, we should process it properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "youngxhui (via GitHub)" <gi...@apache.org>.
youngxhui commented on PR #226:
URL: https://github.com/apache/skywalking-go/pull/226#issuecomment-3016876101
> Please update the CHANGES.md file.
The documentation has been updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add recover to goroutine to prevent unexpected panics [skywalking-go]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #226:
URL: https://github.com/apache/skywalking-go/pull/226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org