You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "Fine0830 (via GitHub)" <gi...@apache.org> on 2026/04/22 03:35:33 UTC
[PR] feat: implement panic diagnostics and FODC crash reporting pipeline [skywalking-banyandb]
Fine0830 opened a new pull request, #1084:
URL: https://github.com/apache/skywalking-banyandb/pull/1084
## Summary
This PR adds an end-to-end panic diagnostics pipeline for BanyanDB and FODC, covering runtime crash output, recovered panic artifacts, breadcrumb/state capture, agent-side crash collection, and proxy-side diagnostics aggregation.
The goal is to make production failures easier to inspect while keeping the runtime path safe and low-overhead during normal operation.
## What Changed
### Panic diagnostics foundation
- add `pkg/panicdiag` as the shared panic diagnostics package
- add recovered panic artifact writing (`panic.json`, `crash.txt`)
- add process-wide panic metrics and reporter hooks
- add runtime crash output bootstrap via `runtime/debug.SetCrashOutput`
- add bounded state dumping and breadcrumb support
- add gRPC breadcrumb interceptors
### BanyanDB integration
- wire panic diagnostics into BanyanDB startup
- expose panic artifact directory configuration through observability/service flags
- extend recovery coverage to more production goroutines and run-group helpers
### FODC agent
- add in-process panic store for recovered panics
- add filesystem watcher and crash analyzer for persisted crash artifacts
- add combined collection provider to merge in-process and filesystem-backed diagnostics
- add diagnostics endpoint exposure from the agent HTTP server
- add cleanup for empty `runtime-crash-*.txt` files on normal exit
- preserve non-empty runtime crash output files for postmortem analysis
### FODC proxy
- add diagnostics aggregation across agents
- add proxy-side diagnostics HTTP/API endpoint
- add metrics/registration flow updates needed for diagnostics collection
### Test coverage and hardening
- add unit and integration coverage for:
- panic artifact persistence
- breadcrumb/state-dump capture
- crash collection and deduplication
- agent/proxy diagnostics flow
- Prometheus export and FODC integration stability
- fix related lint issues and flaky test behavior uncovered while adding the diagnostics pipeline
Signed-off-by: Qiuxia Fan <qi...@apache.org>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement panic diagnostics and FODC crash reporting pipeline [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #1084:
URL: https://github.com/apache/skywalking-banyandb/pull/1084#discussion_r3145978187
##########
pkg/panicdiag/artifact.go:
##########
@@ -0,0 +1,144 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package panicdiag
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+ "time"
+)
+
+const (
+ panicRecordFileName = "panic.json"
+ crashTextFileName = "crash.txt"
+)
+
+// ArtifactWriter writes panic artifacts to disk.
+type ArtifactWriter struct {
+ nowFn func() time.Time
+ pidFn func() int
+ rootDir string
+}
+
+// NewArtifactWriter returns a new ArtifactWriter.
+func NewArtifactWriter(rootDir string) *ArtifactWriter {
+ return &ArtifactWriter{
+ rootDir: rootDir,
+ nowFn: time.Now,
+ pidFn: os.Getpid,
+ }
+}
+
+// Write persists the given panic record and returns the artifact directory.
+func (aw *ArtifactWriter) Write(record *PanicRecord) (string, error) {
+ if aw == nil {
+ return "", fmt.Errorf("artifact writer is nil")
+ }
+ if record == nil {
+ return "", fmt.Errorf("panic record is nil")
+ }
+ if aw.rootDir == "" {
+ return "", fmt.Errorf("artifact root is empty")
+ }
+
+ if err := os.MkdirAll(aw.rootDir, 0o755); err != nil {
+ return "", fmt.Errorf("create artifact root: %w", err)
+ }
+
+ recordedAt := record.OccurredAt
+ if recordedAt.IsZero() {
+ recordedAt = aw.nowFn().UTC()
+ record.OccurredAt = recordedAt
+ }
+
+ artifactDir := filepath.Join(aw.rootDir, aw.dirName(record.Component, recordedAt))
+ if err := os.MkdirAll(artifactDir, 0o755); err != nil {
+ return "", fmt.Errorf("create artifact dir: %w", err)
+ }
+
+ summaryPath := filepath.Join(artifactDir, crashTextFileName)
+ summary := buildCrashSummary(record)
+ if writeErr := os.WriteFile(summaryPath, []byte(summary), 0o600); writeErr != nil {
+ return "", fmt.Errorf("write crash summary: %w", writeErr)
+ }
Review Comment:
**C1: `panic.json` is never written; breadcrumbs / metadata / state-dump status are lost on disk**
The constant `panicRecordFileName = "panic.json"` is declared at line 29 but `Write` only persists `crash.txt` here. `buildCrashSummary` (lines 142-156) emits only `OccurredAt / Component / Recovered / Panic / Stack` — **no `Breadcrumbs`, no `ProcessMetadata`, no `StateDump` status**.
The reader in `collections.go` (`readCollection`) is already coded to prefer `panic.json` and silently fall back to text-parsing `crash.txt` when it's absent. Today that fallback is always taken, so every postmortem read of an artifact — the actual production scenario, where the process that panicked may already be gone — produces a `PanicRecord` with `Breadcrumbs == nil`. The headline benefit of the design (breadcrumbs survive into the artifact) does not hold.
**Suggested fix** — persist the structured record first (source of truth), then the human-readable summary as best-effort:
```go
// 1) Structured record — the source of truth for postmortem readers.
recordPath := filepath.Join(artifactDir, panicRecordFileName)
recordBytes, marshalErr := json.MarshalIndent(record, "", " ")
if marshalErr != nil {
return "", fmt.Errorf("marshal panic record: %w", marshalErr)
}
if writeErr := os.WriteFile(recordPath, recordBytes, 0o600); writeErr != nil {
return "", fmt.Errorf("write panic record: %w", writeErr)
}
// 2) Human-readable summary. Best-effort — don't fail the artifact if this errors.
summaryPath := filepath.Join(artifactDir, crashTextFileName)
if writeErr := os.WriteFile(summaryPath, []byte(buildCrashSummary(record)), 0o600); writeErr != nil {
return artifactDir, fmt.Errorf("write crash summary: %w", writeErr)
}
```
Two follow-ups in adjacent files:
- `collections.go` `readCollection`: prefer `panic.json` when present and treat unmarshal failure as a hard error rather than silently falling back to the text parser.
- `collections.go` `PruneArtifacts`: recognize a directory by `panic.json` **or** `crash.txt`, so a partial write doesn't make a directory invisible to LRU eviction.
A test that proves the fix and would have caught the regression:
```go
func TestArtifactWriter_PersistsBreadcrumbs(t *testing.T) {
dir := t.TempDir()
aw := NewArtifactWriter(dir)
rec := &PanicRecord{
Component: "test",
PanicValue: "boom",
Breadcrumbs: []Breadcrumb{{Stage: "handle", Component: "grpc"}},
ProcessMetadata: map[string]string{"pod": "p1"},
}
out, err := aw.Write(rec)
require.NoError(t, err)
cols, err := ListCollections(dir)
require.NoError(t, err)
require.Len(t, cols, 1)
require.Equal(t, filepath.Base(out), cols[0].ArtifactDir)
require.Equal(t, "handle", cols[0].Record.Breadcrumbs[0].Stage)
require.Equal(t, "p1", cols[0].Record.ProcessMetadata["pod"])
}
```
##########
pkg/run/goroutine.go:
##########
@@ -0,0 +1,39 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package run
+
+import (
+ "context"
+
+ "github.com/apache/skywalking-banyandb/pkg/logger"
+ "github.com/apache/skywalking-banyandb/pkg/panicdiag"
+)
+
+// Go launches fn in a new goroutine protected by panic recovery.
+// Panics are intercepted, and the panic value and localized stack trace are logged.
+// When a panic counter is configured via RecoveryOptions.Counter or
+// panicdiag.SetDefaultPanicCounter, banyandb_panic_total is incremented with
+// component as the label.
+func Go(ctx context.Context, component string, log *logger.Logger, fn func(context.Context)) {
+ panicdiag.GoWithRecovery(ctx, panicdiag.RecoveryOptions{
+ Component: component,
+ Logger: log,
+ }, nil, func(ctxPtr *context.Context) {
+ fn(*ctxPtr)
+ })
Review Comment:
**C3: `run.Go` discards the ctx pointer — breadcrumbs added inside `fn` never reach the recovery defer**
`WithRecovery` reads breadcrumbs from the outer `ctx` variable in its defer (`pkg/panicdiag/recovery.go:50`). The whole point of `fn func(*context.Context)` is so the body can do `*ctxPtr = panicdiag.WithBreadcrumb(*ctxPtr, ...)` and have the defer see the new chain.
This wrapper dereferences once and passes a value-copied ctx to `fn`:
```go
}, nil, func(ctxPtr *context.Context) {
fn(*ctxPtr)
})
```
Any `WithBreadcrumb(innerCtx, ...)` call inside `fn` builds a chain that lives only on `fn`'s stack frame; the outer `ctx` never updates; the defer reads zero breadcrumbs. The only existing escape hatch is the `mutableBreadcrumbStore` installed by gRPC interceptors — but plain `run.Go` callers (which are most of the new wrapping in this PR: `banyand/queue/pub/*`, `banyand/measure/topn.go`, `banyand/internal/storage/*`, `banyand/trace/streaming_pipeline.go`, etc.) get neither.
Combined with C1 (`panic.json` never written), this means breadcrumbs are dropped on both the in-process and on-disk paths for almost every panic site this PR adds. The headline feature is null on the way out for these call sites.
**Suggested fix (Option A) — install a mutable store inside `run.Go` automatically.** One-line change, no caller-API churn:
```go
func Go(ctx context.Context, component string, log *logger.Logger, fn func(context.Context)) {
panicdiag.GoWithRecovery(ctx, panicdiag.RecoveryOptions{
Component: component,
Logger: log,
}, nil, func(ctxPtr *context.Context) {
// Install a mutable breadcrumb store so breadcrumbs added inside
// fn are visible to the recovery defer (which reads *ctxPtr after
// fn returns or panics).
*ctxPtr = panicdiag.WithMutableBreadcrumbs(*ctxPtr)
fn(*ctxPtr)
})
}
```
This relies on C2 being fixed as well — without the mutex, `run.Go` workers that share a parent's mutable store will race.
A test that proves the fix and would have caught the regression:
```go
func TestGo_RecoveryCapturesBreadcrumbsAddedInFn(t *testing.T) {
var capturedRecord *panicdiag.PanicRecord
panicdiag.SetDefaultReporter(func(_ context.Context, r panicdiag.RecoveryResult) {
capturedRecord = r.Record
})
t.Cleanup(func() { panicdiag.SetDefaultReporter(nil) })
done := make(chan struct{})
run.Go(context.Background(), "test", logger.GetLogger("test"),
func(ctx context.Context) {
defer close(done)
panicdiag.WithBreadcrumb(ctx, "stage-A", "component", nil)
panicdiag.WithBreadcrumb(ctx, "stage-B", "component", nil)
panic("boom")
})
<-done
require.NotNil(t, capturedRecord)
require.Len(t, capturedRecord.Breadcrumbs, 2)
require.Equal(t, "stage-A", capturedRecord.Breadcrumbs[0].Stage)
require.Equal(t, "stage-B", capturedRecord.Breadcrumbs[1].Stage)
}
```
##########
pkg/panicdiag/breadcrumb.go:
##########
@@ -0,0 +1,145 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package panicdiag
+
+import (
+ "context"
+ "time"
+)
+
+type breadcrumbContextKey struct{}
+
+// mutableBreadcrumbContextKey is the context key for a mutable breadcrumb store.
+// When present it takes precedence over the immutable linked-list path so that
+// gRPC recovery interceptors — which capture the context at interceptor entry,
+// before the handler runs — can still read breadcrumbs appended inside the handler.
+type mutableBreadcrumbContextKey struct{}
+
+// mutableBreadcrumbStore holds breadcrumbs that can be appended without creating
+// a new context value. A pointer to this struct is stored in the context so that
+// mutation is visible through the original context reference.
+type mutableBreadcrumbStore struct {
+ breadcrumbs []Breadcrumb
+}
Review Comment:
**C2: `mutableBreadcrumbStore` is unsynchronized — concurrent `WithBreadcrumb` calls race**
The slice is mutated by `WithBreadcrumb` (append at line 81) and read by `BreadcrumbsFromContext` (`copy` at line 116) without any lock. Once a gRPC interceptor (`grpcinterceptors.go:24-33`) installs the store, the same `*mutableBreadcrumbStore` is reachable from every goroutine that derives a context from the request — including the many `run.Go` workers spawned in handlers throughout this PR. Concurrent `append` tears the slice header; `go test -race` will fire as soon as any test exercises more than one goroutine on a shared context.
The reader runs from inside the panic-recovery defer (`recovery.go:50`). A torn slice there can re-panic and bypass the artifact write entirely — neither breadcrumbs nor the recover-and-log path land.
**Suggested fix** — add a `sync.RWMutex` to the store and use it on both paths. Hot only on panic; on the happy path the cost is one uncontended `Lock/Unlock` per breadcrumb (negligible vs. the existing per-call `cloneStringMap`).
```go
type mutableBreadcrumbStore struct {
mu sync.RWMutex
breadcrumbs []Breadcrumb
}
```
In `WithBreadcrumb`:
```go
if store, ok := ctx.Value(mutableBreadcrumbContextKey{}).(*mutableBreadcrumbStore); ok {
store.mu.Lock()
if len(store.breadcrumbs) < maxBreadcrumbDepth {
store.breadcrumbs = append(store.breadcrumbs, Breadcrumb{
Time: nowBreadcrumbTime(),
Stage: stage,
Component: component,
Fields: cloneStringMap(fields),
})
}
store.mu.Unlock()
return ctx
}
```
In `BreadcrumbsFromContext`:
```go
if store, ok := ctx.Value(mutableBreadcrumbContextKey{}).(*mutableBreadcrumbStore); ok {
store.mu.RLock()
defer store.mu.RUnlock()
if len(store.breadcrumbs) == 0 { return nil }
result := make([]Breadcrumb, len(store.breadcrumbs))
copy(result, store.breadcrumbs)
for i := range result {
result[i].Fields = cloneStringMap(result[i].Fields)
}
return result
}
```
A focused race test (must fail on the current code under `-race`):
```go
func TestMutableBreadcrumbs_ConcurrentAppendSafe(t *testing.T) {
ctx := WithMutableBreadcrumbs(context.Background())
var wg sync.WaitGroup
for i := 0; i < 16; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
for j := 0; j < 100; j++ {
WithBreadcrumb(ctx, "stage", "component", map[string]string{"i": strconv.Itoa(i)})
_ = BreadcrumbsFromContext(ctx)
}
}(i)
}
wg.Wait()
if got := BreadcrumbsFromContext(ctx); len(got) > maxBreadcrumbDepth {
t.Fatalf("breadcrumb depth exceeded cap: %d", len(got))
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement panic diagnostics and FODC crash reporting pipeline [skywalking-banyandb]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1084:
URL: https://github.com/apache/skywalking-banyandb/pull/1084#issuecomment-4296180226
## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `52.38880%` with `289 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 31.24%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`077c219`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/077c219706b9b94e90b1716cfdc46c76f7defd12?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 216 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [pkg/panicdiag/artifact.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fartifact.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9hcnRpZmFjdC5nbw==) | 46.78% | [39 Missing and 19 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/collections.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fcollections.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9jb2xsZWN0aW9ucy5nbw==) | 51.92% | [34 Missing and 16 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/crashoutput.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fcrashoutput.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9jcmFzaG91dHB1dC5nbw==) | 51.25% | [30 Missing and 9 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/breadcrumb.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fbreadcrumb.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9icmVhZGNydW1iLmdv) | 50.00% | [29 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/recovery.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Frecovery.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9yZWNvdmVyeS5nbw==) | 76.84% | [12 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/metrics.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fmetrics.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9tZXRyaWNzLmdv) | 42.85% | [15 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/timestamp/scheduler.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Ftimestamp%2Fscheduler.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3RpbWVzdGFtcC9zY2hlZHVsZXIuZ28=) | 0.00% | [16 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/grpcinterceptors.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fgrpcinterceptors.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9ncnBjaW50ZXJjZXB0b3JzLmdv) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/panicdiag/state.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Fpanicdiag%2Fstate.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3BhbmljZGlhZy9zdGF0ZS5nbw==) | 72.34% | [9 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pkg/run/run.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&filepath=pkg%2Frun%2Frun.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3J1bi9ydW4uZ28=) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [3 more](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
> :exclamation: There is a different number of reports uploaded between BASE (3530dd9) and HEAD (077c219). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (3530dd9) | HEAD (077c219) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #1084 +/- ##
===========================================
- Coverage 45.97% 31.24% -14.74%
===========================================
Files 328 135 -193
Lines 55505 16127 -39378
===========================================
- Hits 25520 5039 -20481
+ Misses 27909 10635 -17274
+ Partials 2076 453 -1623
```
| [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.24% <52.38%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1084?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement panic diagnostics and FODC crash reporting pipeline [skywalking-banyandb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1084:
URL: https://github.com/apache/skywalking-banyandb/pull/1084#discussion_r3121431125
##########
pkg/timestamp/scheduler.go:
##########
@@ -227,9 +229,9 @@ func (t *task) run() {
resultCh := make(chan bool, 1)
timeoutCh := t.clock.Timer(5 * time.Minute).C
- go func() {
- resultCh <- t.action(now, t.l)
- }()
+ run.Go(context.Background(), "scheduler-action-"+t.name, t.l, func(ctx context.Context) {
+ resultCh <- t.action(ctx, now, t.l)
+ })
Review Comment:
The scheduler action goroutine passes context.Background() into SchedulerAction, so the action ctx is never cancelled when the scheduler/task is closed (despite the SchedulerAction doc saying it is). This also means long-running actions won't be signaled to stop on Scheduler.Close() or task.close(). Consider deriving a context that's cancelled on t.closer.CloseNotify() (or storing a task-scoped context cancelled in t.close()), and pass that into run.Go / action instead of context.Background().
##########
fodc/proxy/internal/diagnostics/aggregator.go:
##########
@@ -0,0 +1,228 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package diagnostics provides functionality for aggregating crash/panic diagnostic data from all agents.
+package diagnostics
+
+import (
+ "context"
+ "fmt"
+ "strings"
+ "sync"
+ "time"
+
+ fodcv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/fodc/v1"
+ "github.com/apache/skywalking-banyandb/fodc/proxy/internal/registry"
+ "github.com/apache/skywalking-banyandb/pkg/logger"
+)
+
+// refreshWait is the time given to agents to respond after a RequestDiagnostics call.
+// The agent sends one gRPC message per crash record synchronously; this window
+// collects all messages that arrive before returning the cached view.
+const refreshWait = 2 * time.Second
+
+// CrashPanicInfo holds the panic details from a crash record.
+type CrashPanicInfo struct {
+ OccurredAt time.Time `json:"occurred_at"`
+ Component string `json:"component"`
+ GoroutineStack string `json:"goroutine_stack,omitempty"`
+ PanicValue string `json:"panic_value"`
+ Recovered bool `json:"recovered"`
+}
+
+// AggregatedCrashRecord enriches a crash collection with agent identity.
+type AggregatedCrashRecord struct {
+ FetchedAt time.Time `json:"fetched_at"`
+ PanicRecord *CrashPanicInfo `json:"panic_record,omitempty"`
+ AgentID string `json:"agent_id"`
+ PodName string `json:"pod_name"`
+ Role string `json:"role"`
+ SourceEndpoint string `json:"source_endpoint"`
+ ArtifactDir string `json:"artifact_dir"`
+ Files []string `json:"files,omitempty"`
+}
+
+// Filter defines filters for diagnostics collection.
+type Filter struct {
+ Role string
+ PodName string
+}
+
+// RequestSender is an interface for requesting crash diagnostics from agents.
+type RequestSender interface {
+ RequestDiagnostics(agentID string) error
+}
+
+// Aggregator aggregates crash diagnostic data from all agents.
+//
+// Unlike the metrics aggregator (which uses a request/response channel per
+// collection cycle), crash records arrive as one gRPC message per record with
+// no end-of-batch signal in the current proto. The aggregator therefore keeps
+// a dedup cache keyed by "agentID::artifactDir". On each HTTP request it
+// triggers a fresh pull from every agent, waits a short window for messages to
+// arrive, then returns the current cache snapshot.
+type Aggregator struct {
+ registry *registry.AgentRegistry
+ grpcService RequestSender
+ // cache maps "agentID::artifactDir" → record for O(1) dedup.
+ cache map[string]*AggregatedCrashRecord
+ log *logger.Logger
+
+ cacheMu sync.RWMutex
+ mu sync.RWMutex
+}
+
+// NewAggregator creates a new Aggregator instance.
+func NewAggregator(registry *registry.AgentRegistry, grpcService RequestSender, log *logger.Logger) *Aggregator {
+ return &Aggregator{
+ registry: registry,
+ grpcService: grpcService,
+ log: log,
+ cache: make(map[string]*AggregatedCrashRecord),
+ }
+}
+
+// SetGRPCService sets the gRPC service for sending diagnostics requests.
+func (a *Aggregator) SetGRPCService(grpcService RequestSender) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ a.grpcService = grpcService
+}
+
+// RemoveAgent removes all cached records for an agent (called on disconnect).
+func (a *Aggregator) RemoveAgent(agentID string) {
+ a.cacheMu.Lock()
+ defer a.cacheMu.Unlock()
+ prefix := agentID + "::"
+ for key := range a.cache {
+ if strings.HasPrefix(key, prefix) {
+ delete(a.cache, key)
+ }
+ }
+}
+
+// ProcessCrashFromAgent stores or updates the cached record for this crash artifact.
+// Each StreamCrashDiagnosticsRequest carries exactly one artifact; dedup is by artifactDir.
+func (a *Aggregator) ProcessCrashFromAgent(agentID string, agentInfo *registry.AgentInfo, req *fodcv1.StreamCrashDiagnosticsRequest) {
+ record := &AggregatedCrashRecord{
+ AgentID: agentID,
+ PodName: agentInfo.AgentIdentity.PodName,
+ Role: agentInfo.AgentIdentity.Role,
+ SourceEndpoint: req.SourceEndpoint,
+ ArtifactDir: req.ArtifactDir,
+ Files: req.Files,
+ }
+ if req.FetchedAt != nil {
+ record.FetchedAt = req.FetchedAt.AsTime()
+ }
+ if req.PanicRecord != nil {
+ info := &CrashPanicInfo{
+ Component: req.PanicRecord.Component,
+ PanicValue: req.PanicRecord.PanicValue,
+ Recovered: req.PanicRecord.Recovered,
+ GoroutineStack: req.PanicRecord.GoroutineStack,
+ }
+ if req.PanicRecord.OccurredAt != nil {
+ info.OccurredAt = req.PanicRecord.OccurredAt.AsTime()
+ }
+ record.PanicRecord = info
+ }
+
+ cacheKey := fmt.Sprintf("%s::%s", agentID, req.ArtifactDir)
+ a.cacheMu.Lock()
+ a.cache[cacheKey] = record
+ a.cacheMu.Unlock()
+}
+
+// CollectDiagnostics triggers a refresh from all matching agents, waits a short
+// window for their records to arrive, then returns the current cache snapshot.
+func (a *Aggregator) CollectDiagnostics(ctx context.Context, filter *Filter) ([]*AggregatedCrashRecord, error) {
+ a.mu.RLock()
+ grpcService := a.grpcService
+ a.mu.RUnlock()
+
+ agents := a.getFilteredAgents(filter)
+
+ // Request a fresh send from each agent. Failures are non-fatal — the agent
+ // may not have established its crash stream yet.
+ for _, agentInfo := range agents {
+ select {
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ default:
+ }
+ if requestErr := grpcService.RequestDiagnostics(agentInfo.AgentID); requestErr != nil {
+ a.log.Info().Err(requestErr).Str("agent_id", agentInfo.AgentID).
+ Msg("Agent does not support crash diagnostics stream, skipping")
Review Comment:
CollectDiagnostics reads a.grpcService under lock but doesn't guard against it being nil. Since NewAggregator is constructed with grpcService=nil in main before SetGRPCService is called, a concurrent /diagnostics request during startup (or any other caller) could nil-deref at grpcService.RequestDiagnostics(...). Add a nil check and either skip requesting (return cached snapshot) or return a clear error when grpcService is unset.
##########
pkg/run/goroutine.go:
##########
@@ -0,0 +1,37 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package run
+
+import (
+ "context"
+
+ "github.com/apache/skywalking-banyandb/pkg/logger"
+ "github.com/apache/skywalking-banyandb/pkg/panicdiag"
+)
+
+// Go launches fn in a new goroutine protected by panic recovery.
+// Panics are intercepted, the panic value and localized stack trace are logged,
+// and banyandb_panic_total is incremented with component as the label.
+func Go(ctx context.Context, component string, log *logger.Logger, fn func(context.Context)) {
Review Comment:
The doc comment claims run.Go "increments banyandb_panic_total" unconditionally, but incPanicCounter is a no-op unless RecoveryOptions.Counter is set or panicdiag.SetDefaultPanicCounter has been called. Please adjust the comment to reflect that the metric is only incremented when a counter is configured, to avoid misleading callers/tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement panic diagnostics and FODC crash reporting pipeline [skywalking-banyandb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1084:
URL: https://github.com/apache/skywalking-banyandb/pull/1084#discussion_r3223294064
##########
pkg/cmdsetup/data.go:
##########
@@ -111,8 +111,11 @@ func newDataCmd(runners ...run.Unit) *cobra.Command {
return err
}
logger.GetLogger().Info().Msg("starting as a data server")
- // Spawn our go routines and wait for shutdown.
- if err := dataGroup.Run(context.WithValue(context.Background(), common.ContextNodeKey, node)); err != nil {
+ // Spawn our go routines and wait for shutdown. The Run context is
+ // derived from SupervisorContext so that a panic recovered anywhere
+ // in the process, including goroutines spawned via run.Go, fires
+ // cancellation here.
+ if err := dataGroup.Run(context.WithValue(SupervisorContext(), common.ContextNodeKey, node)); err != nil {
logger.GetLogger().Error().Err(err).Stack().Str("name", dataGroup.Name()).Msg("Exit")
os.Exit(-1)
}
Review Comment:
This RunE path calls os.Exit(-1) on error, which bypasses Cobra’s PersistentPostRunE. Since root.go now uses PersistentPostRunE to call ShutdownSupervisor() (draining the panic artifact sink), os.Exit here can drop queued artifacts on error exits. Prefer returning the error (letting Cobra unwind and run post-run hooks) or explicitly calling ShutdownSupervisor() before exiting.
##########
pkg/timestamp/scheduler.go:
##########
@@ -39,9 +40,10 @@ var (
ErrTaskDuplicated = errors.New("the task is duplicated")
)
-// SchedulerAction is an executable when a trigger is fired
-// now is the trigger time, logger has a context indicating the task's identity.
-type SchedulerAction func(now time.Time, logger *logger.Logger) bool
+// SchedulerAction is an executable when a trigger is fired.
+// ctx is canceled when the scheduler is closed, now is the trigger time,
+// and logger carries the task's identity.
+type SchedulerAction func(ctx context.Context, now time.Time, logger *logger.Logger) bool
Review Comment:
SchedulerAction’s doc says the passed ctx is canceled when the scheduler is closed, but Register stores the caller ctx in task.parentCtx and Close() only closes t.closer (it never cancels parentCtx). As a result, action goroutines started via run.GoWithSignal may keep running after Scheduler.Close(), and callers relying on ctx cancellation won’t observe shutdown. Consider deriving a task context via context.WithCancel (or using t.closer.Ctx()) and canceling it in task.close(), and update run.Go / GoWithSignal to use that derived ctx.
##########
pkg/cmdsetup/standalone.go:
##########
@@ -118,8 +118,11 @@ func newStandaloneCmd(runners ...run.Unit) *cobra.Command {
return err
}
logger.GetLogger().Info().Msg("starting as a standalone server")
- // Spawn our go routines and wait for shutdown.
- if err := standaloneGroup.Run(context.WithValue(context.Background(), common.ContextNodeKey, nodeID)); err != nil {
+ // Spawn our go routines and wait for shutdown. The Run context is
+ // derived from SupervisorContext so that a panic recovered anywhere
+ // in the process, including goroutines spawned via run.Go, fires
+ // cancellation here.
+ if err := standaloneGroup.Run(context.WithValue(SupervisorContext(), common.ContextNodeKey, nodeID)); err != nil {
logger.GetLogger().Error().Err(err).Stack().Str("name", standaloneGroup.Name()).Msg("Exit")
os.Exit(-1)
}
Review Comment:
This RunE path calls os.Exit(-1) on error, which bypasses Cobra’s PersistentPostRunE. Since root.go now uses PersistentPostRunE to call ShutdownSupervisor() (draining the panic artifact sink), os.Exit here can drop queued artifacts on error exits. Prefer returning the error (letting Cobra unwind and run post-run hooks) or explicitly calling ShutdownSupervisor() before exiting.
##########
pkg/cmdsetup/liaison.go:
##########
@@ -157,9 +160,9 @@ func newLiaisonCmd(runners ...run.Unit) *cobra.Command {
return err
}
logger.GetLogger().Info().Msg("starting as a liaison server")
- ctx = context.WithValue(ctx, common.ContextNodeKey, node)
+ runCtx = context.WithValue(runCtx, common.ContextNodeKey, node)
// Spawn our go routines and wait for shutdown.
- if err := liaisonGroup.Run(ctx); err != nil {
+ if err := liaisonGroup.Run(runCtx); err != nil {
logger.GetLogger().Error().Err(err).Stack().Str("name", liaisonGroup.Name()).Msg("Exit")
os.Exit(-1)
}
Review Comment:
This RunE path calls os.Exit(-1) on error, which bypasses Cobra’s PersistentPostRunE. Since root.go now uses PersistentPostRunE to call ShutdownSupervisor() (draining the panic artifact sink), os.Exit here can drop queued artifacts on error exits. Prefer returning the error (letting Cobra unwind and run post-run hooks) or explicitly calling ShutdownSupervisor() before exiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement panic diagnostics and FODC crash reporting pipeline [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #1084:
URL: https://github.com/apache/skywalking-banyandb/pull/1084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org