You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "otokaze (via GitHub)" <gi...@apache.org> on 2026/05/26 05:08:15 UTC
[I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
otokaze opened a new issue, #13885:
URL: https://github.com/apache/skywalking/issues/13885
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
Go Agent (apache/skywalking-go)
### What happened
go agent data race.
### What you expected to happen
unexpected fault address
### How to reproduce
skywalking-go@main (latest commit, non-release version for PostgreSQL support)
### Anything else
unexpected fault address 0x1f28cf0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x1f28cf0 pc=0x7ef153]
goroutine 23 gp=0xc0002d94a0 m=10 mp=0xc0105d1008 [running]:
runtime.throw({0x1d0557a?, 0xc00dbdac40?})
/usr/local/go/src/runtime/panic.go:1094 +0x48 fp=0xc020f8e710 sp=0xc020f8e6e0 pc=0x4801a8
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:939 +0x26c fp=0xc020f8e770 sp=0xc020f8e710 pc=0x48270c
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc00027c608, {0x1bd?}, {0x0?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:128 +0x2f3 fp=0xc020f8e7e8 sp=0xc020f8e770 pc=0x7ef153
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0xc00048e680?, {0xc01b1bf800?}, {0xc0?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:60 +0x6f fp=0xc020f8e820 sp=0xc020f8e7e8 pc=0x7eee0f
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0x200703e10ed6900?}, 0xc000d145e0, {0x2d?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/codec_field.go:414 +0x4d fp=0xc020f8e870 sp=0xc020f8e820 pc=0x7c048d
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002a2ad8, {0xc0004ee818?}, {0x31?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:111 +0x245 fp=0xc020f8e8e8 sp=0xc020f8e870 pc=0x7ef0a5
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x1bd?, {0x441707?}, {0x0?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:60 +0x6f fp=0xc020f8e920 sp=0xc020f8e8e8 pc=0x7eee0f
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc0002a2808?}, 0xc0004ee818, {0x1c?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/codec_field.go:414 +0x4d fp=0xc020f8e970 sp=0xc020f8e920 pc=0x7c048d
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002a2808, {0x76af45?}, {0xf8?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:111 +0x245 fp=0xc020f8e9e8 sp=0xc020f8e970 pc=0x7ef0a5
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0xc020f8ea58?, {0x492e45?}, {0x80?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:60 +0x6f fp=0xc020f8ea20 sp=0xc020f8e9e8 pc=0x7eee0f
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x0?, {{}, {0x21ccaf8?, 0xc021fe2990?}, 0xcd?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:41 +0x3e fp=0xc020f8ea48 sp=0xc020f8ea20 pc=0x7eed3e
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x21ccaf8?, 0xc021fe2990?}, 0xa8?})
<autogenerated>:1 +0x35 fp=0xc020f8ea78 sp=0xc020f8ea48 pc=0x81d235
google.golang.org/protobuf/proto.MarshalOptions.size({{}, 0x90?, 0x29?, 0xfe?}, {0x21ccaf8, 0xc021fe2990})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/proto/size.go:43 +0x82 fp=0xc020f8eaf0 sp=0xc020f8ea78 pc=0x76ec02
google.golang.org/protobuf/proto.MarshalOptions.Size({{}, 0xe0?, 0x6c?, 0xc1?}, {0x21ab880?, 0xc021fe2990?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/proto/size.go:34 +0x4c fp=0xc020f8eb20 sp=0xc020f8eaf0 pc=0x76eb2c
google.golang.org/protobuf/proto.Size(...)
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/proto/size.go:20
google.golang.org/grpc/encoding/proto.(*codecV2).Marshal(0x0?, {0x1c16ce0?, 0xc021fe2990?})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/encoding/proto/proto.go:51 +0x7a fp=0xc020f8ebd0 sp=0xc020f8eb20 pc=0x173ea7a
google.golang.org/grpc.encode({0x7fee8fe730e0?, 0x30bb1a0?}, {0x1c16ce0?, 0xc021fe2990?})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/rpc_util.go:734 +0x4a fp=0xc020f8ec50 sp=0xc020f8ebd0 pc=0x176770a
google.golang.org/grpc.prepareMsg({0x1c16ce0?, 0xc021fe2990?}, {0x7fee8fe730e0?, 0x30bb1a0?}, {0x0, 0x0}, {0x0, 0x0}, {0x21b7010, 0xc00042c100})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/stream.go:1858 +0xe5 fp=0xc020f8ece8 sp=0xc020f8ec50 pc=0x1782565
google.golang.org/grpc.(*clientStream).SendMsg(0xc01055ea00, {0x1c16ce0, 0xc021fe2990})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/stream.go:924 +0x19f fp=0xc020f8ef00 sp=0xc020f8ece8 pc=0x177ae1f
github.com/apache/skywalking-go/protocols/collect/language/agent/v3.(*traceSegmentReportServiceCollectClient).Send(0x21bae68?, 0x21baea0?)
/go/pkg/mod/github.com/apache/skywalking-go@v0.6.1-0.20260502031559-052bf4b42c13/protocols/collect/language/agent/v3/Tracing_grpc.pb.go:61 +0x25 fp=0xc020f8ef28 sp=0xc020f8ef00 pc=0x17ca4e5
github.com/apache/skywalking-go/agent/reporter.(*gRPCReporter).initSendPipeline.func1()
grpc.go:212 +0x226 fp=0xc020f8efe0 sp=0xc020f8ef28 pc=0x17e10a6
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc020f8efe8 sp=0xc020f8efe0 pc=0x4885a1
created by github.com/apache/skywalking-go/agent/reporter.(*gRPCReporter).initSendPipeline in goroutine 1
grpc.go:189 +0x5f
### Are you willing to submit a pull request to fix on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13885:
URL: https://github.com/apache/skywalking/issues/13885#issuecomment-4565093425
Then, I am closing this. Feel free to open a new one if there are new found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
Posted by "otokaze (via GitHub)" <gi...@apache.org>.
otokaze commented on issue #13885:
URL: https://github.com/apache/skywalking/issues/13885#issuecomment-4552630434
> Hi, [@otokaze](https://github.com/otokaze). Could you help build a quick demo about using the `PostgreSQL` framework with Crash? I want to check why this will happen in this framework.
Thank you for your reply!
This bug occurs intermittently, roughly once a day, and I currently don't have a reliable way to reproduce it. The stack trace provided above is from when the crash occurred. Is there any temporary workaround I can apply in the meantime? Should I add a lock during serialization?
I'm currently using skywalking-go@052bf4b42c13 (the version that includes PostgreSQL support). If this specific commit is unstable, would you recommend I manually port the pgsql support onto the latest stable release myself to avoid this issue? To be honest, I'd strongly prefer not to go down that route if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on issue #13885:
URL: https://github.com/apache/skywalking/issues/13885#issuecomment-4555213146
Hi @otokaze. I have updated the skywalking-go project. Could you help recompile it with the [latest Go agent](https://github.com/apache/skywalking-go/tree/b59eb7654b9219b10592629e35b6993aa87f0812)? Please let me know if it works or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #13885: [Bug] Service crash caused by Go agent unexpected fault address
URL: https://github.com/apache/skywalking/issues/13885
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on issue #13885:
URL: https://github.com/apache/skywalking/issues/13885#issuecomment-4544814603
Hi, @otokaze. Could you help build a quick demo about using the `PostgreSQL` framework with Crash? I want to check why this will happen in this framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Service crash caused by Go agent unexpected fault address [skywalking]
Posted by "otokaze (via GitHub)" <gi...@apache.org>.
otokaze commented on issue #13885:
URL: https://github.com/apache/skywalking/issues/13885#issuecomment-4565050541
> Hi [@otokaze](https://github.com/otokaze). I have updated the skywalking-go project. Could you help recompile it with the [latest Go agent](https://github.com/apache/skywalking-go/tree/b59eb7654b9219b10592629e35b6993aa87f0812)? Please let me know if it works or not.
Thx!
We have conducted extensive stress testing in our staging environment and confirmed that this issue no longer occurs. Thank you to the technical team for your incredible support!
This has greatly boosted our confidence in continuing to use SkyWalking in our production environment. We will keep you updated on its production status moving forward.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org