You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "Felix-wave (via GitHub)" <gi...@apache.org> on 2026/05/06 07:48:39 UTC
[I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Felix-wave opened a new issue, #13861:
URL: https://github.com/apache/skywalking/issues/13861
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues. (Different from the related #13860 — that one is in the trace **write** path and is recovered by the gRPC interceptor; this one is in the trace **merge/read** path and crashes the process.)
### Apache SkyWalking Component
BanyanDB
### What happened
After upgrading from `apache/skywalking-banyandb:0.9.0` to `0.10.1` (with OAP 10.4.0), BanyanDB **crashes the process** every ~7-8 minutes with:
```
panic: offset 1400877 must be equal to bytesRead 1400490
```
Unlike the timestamp-ordering panic in #13860 (which is recovered by `grpc-middleware`), this one fires from a **background `mergeLoop` goroutine** that is not wrapped by recovery, so the process exits and the pod restarts.
#### Full stack
```
goroutine 3900 [running]:
github.com/apache/skywalking-banyandb/pkg/logger.Panicf(...)
github.com/apache/skywalking-banyandb/banyand/trace.(*partMergeIter).mustReadRaw(0xc001ac4000, 0xc002d716b8, 0xc001ac4118)
/mnt/d/skywalking-banyandb/banyand/trace/part_iter.go:359 +0xf5
github.com/apache/skywalking-banyandb/banyand/trace.(*blockReader).mustReadRaw(...)
/mnt/d/skywalking-banyandb/banyand/trace/block_reader.go:263
github.com/apache/skywalking-banyandb/banyand/trace.mergeBlocks(...)
/mnt/d/skywalking-banyandb/banyand/trace/merger.go:421 +0x79e
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeParts(...)
/mnt/d/skywalking-banyandb/banyand/trace/merger.go:344 +0x42a
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergePartsThenSendIntroduction(...)
/mnt/d/skywalking-banyandb/banyand/trace/merger.go:118 +0x145
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeSnapshot(...)
/mnt/d/skywalking-banyandb/banyand/trace/merger.go:104 +0x125
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeLoop.func1(...)
/mnt/d/skywalking-banyandb/banyand/trace/merger.go:78 +0x1f9
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeLoop(...)
/mnt/d/skywalking-banyandb/banyand/trace/merger.go:90 +0x271
created by github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).startLoop in goroutine 157
/mnt/d/skywalking-banyandb/banyand/trace/tstable.go:130 +0x246
```
#### Source location (apache/skywalking-banyandb v0.10.1)
`banyand/trace/part_iter.go:354-365`:
```go
func (pmi *partMergeIter) mustReadRaw(r *rawBlock, bm *blockMetadata) {
r.bm = bm
// spans
if bm.spans != nil && bm.spans.size > 0 {
// Validate the reader is aligned to the expected offset
if bm.spans.offset != pmi.seqReaders.spans.bytesRead {
logger.Panicf("offset %d must be equal to bytesRead %d", bm.spans.offset, pmi.seqReaders.spans.bytesRead)
}
...
}
...
}
```
So the merger sequentially reads spans from `seqReaders.spans`, and a per-block `bm.spans.offset` is expected to match how far the `seqReader` has advanced (`bytesRead`). When they diverge — by 387 bytes in our sample — the merger panics. The same pattern (`offset must be equal to bytesRead`) appears at:
- `banyand/trace/block.go:196` (tag metadata)
- `banyand/trace/block.go:329` (span data)
- `banyand/internal/sidx/block.go`
- `banyand/measure/block.go`
- `banyand/stream/block.go`
So the invariant is repeated across the new (0.10) trace storage engine.
### Cadence and impact
In our cluster, BanyanDB pod restarted **126 times in 17 hours** = roughly once every 8 minutes. Every time, OAP loses connection to BanyanDB and hot-loops crash too (~148 OAP restarts in the same window). Net effect: rolling availability — every ~8 minutes there is a 1-2 minute window where ingestion and queries fail.
For comparison, on 0.9.0 the only panic we saw fired ~once every 28 minutes. **0.10.1 is significantly less stable on our workload, primarily because of this new panic in the merger.**
### What you expected to happen
The merger should not panic on what is clearly a corrupted or out-of-sync block metadata. Reasonable options (maintainers know best):
1. **Skip the offending block** with a warning instead of `Panicf` — at minimum, contain the blast radius to one block instead of restarting the whole DB.
2. **Restart the seqReader** to the offset declared in `bm.spans.offset` (or vice versa) when divergence is detected — assumes the metadata is the source of truth.
3. **Fail the merge of the affected part** but keep the process running and let retention/cleanup eventually drop the corrupted part.
### How to reproduce
Steady-state SkyWalking deployment, OAP forwarding traces to standalone BanyanDB. We see this on:
- BanyanDB: `apache/skywalking-banyandb:0.10.1`
- SkyWalking OAP: `apache/skywalking-oap-server:10.4.0`
- ~30+ Java services, `apache-skywalking-java-agent` 9.5.0, JDK 21
- Standalone BanyanDB on Kubernetes (Aliyun ACK), `--trace-root-path=/data/trace`
- 51 GB cumulative on disk after 17h of ingest (stream 38.5G + trace 12.9G + measure 26M)
The very first occurrence happens within ~30 minutes of starting fresh (after fully wiping `/data` and letting OAP recreate schemas). After that, panic cadence stabilizes at ~8 minutes.
### Anything else
This bug is in the new trace storage engine introduced by #713 in 0.10.0; we did not see this panic on 0.9.0 (which uses the older trace path).
We have already reported the related — but distinct — timestamp-ordering panic in the **write** path as #13860 (recoverable, not crashing the process). Filing this one separately because the failure mode (background merge goroutine, no recovery, full process exit) is different and arguably more disruptive.
Happy to gather more samples (full stack traces over time, sample part dumps if a tool exists, sysrq dumps, anything) on request.
### Are you willing to submit a pull request to fix on your own
- [ ] Yes, I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4405733069
0.10.3 is planned, should be in next week with cherry pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "Felix-wave (via GitHub)" <gi...@apache.org>.
Felix-wave commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4404032233
Quick update with strong evidence that **PR #1114 (the trace-write timestamp panic fix) was the missing piece** for stability on our workload.
## Setup
I built a custom image that takes `v0.10.x` (which is already at v0.10.2) and cherry-picks the `e810640` patch (PR #1114). Then I wiped both `/data` directories completely (as recommended by your torn-write analysis) and brought up KL Prod on this image.
```
banyandb image: skywalking-banyandb v0.10.x + cherry-pick e810640 (PR #1114)
OAP image: apache/skywalking-oap-server:10.4.0 (unchanged)
disk: wiped clean before start
```
## Result: 12 hours, zero panics, zero restarts
```
KL Prod banyandb (12h after fresh start):
pod restartCount = 0
#13860 timestamp panic = 0
#13861 offset/bytesRead panic = 0
total panic events = 0
errors = 0 (excluding the 'watcher channel full' info noise)
data ingested = stream 46.6 GB / trace 98.1 GB / measure 55.5 MB
```
For comparison on the same cluster / workload over similar windows:
| Build | Run length | Pod restarts | #13860 timestamp panic | #13861 offset panic |
| --- | --- | --- | --- | --- |
| 0.9.0 + OAP 10.3.0 | 17 h | 17 h / ~28 min ≈ 36 (we measured ~406 over 7 days) | many (process exit) | n/a (different storage engine) |
| 0.10.1 + OAP 10.4.0 (fresh disk) | 17 h | 41 | many | 41 (process exit) |
| 0.10.2 + OAP 10.4.0 (fresh disk) | 80 min before SG hit a backgroud-goroutine timestamp panic that wasn't caught by gRPC recovery | 1 | several | 0 (good — race fix in 0.10.2 worked) |
| **v0.10.x + e810640 (this build) + fresh disk** | **12 h and counting** | **0** | **0** | **0** |
So in our environment **PR #1114 was the load-bearing fix**: 0.10.2's race-shutdown fix on its own was enough to suppress #13861 (the merger-loop panic) on a clean disk, but **#13860 was still firing from background goroutines (introducer/persister/merger of bluge index, measure flush/merge) where there is no gRPC recovery interceptor to catch it**, so a single timestamp panic in the wrong goroutine still took the process down — that's exactly what we saw on SG.
With #1114 landed, those background paths stop generating the panic in the first place, and the cluster runs cleanly.
## Operational confirmation
Functional probe via the OAP GraphQL endpoint while running on this image:
- service registry visible: 23 services including `auto-creator-backend-prod`, `data-api-hub-*`, `spider-manager-service-*`
- `service_resp_time` (avg) returns sensible numbers (~36-68 ms range, in line with what we observed on previous storage backends)
- `service_cpm` returns expected per-minute call counts (111-302 / min)
## Cherry-pick to v0.10.x?
Given the impact, would it be worth cherry-picking #1114 onto v0.10.x and cutting a 0.10.3? We're happy to keep running our own build until upstream catches up, but I expect anyone else hitting this on a multi-agent SkyWalking deployment will see the same instability — the panic was firing every ~4 minutes for us before the fix.
Thanks again to you and @wu-sheng for the fast turnaround on this — really appreciate the in-depth root-cause work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "Felix-wave (via GitHub)" <gi...@apache.org>.
Felix-wave commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4394533813
Brilliant root-cause work, @hanahmily — the torn-`spans.bin` reproducer matches our panic shape character-for-character, and the audit in #13862 lines up with the symptoms (perpetuates across restarts, hits all four storage engines that share the same merge writer). The two-phase commit + quarantine plan looks right.
Here are the diagnostics you asked for. Spoiler: I don't think OOMKill or eviction is the first-cause tear in our setup, but I have an alternative hypothesis below worth considering.
### 1. Pod exit history
Both clusters' BanyanDB pods exit cleanly via `panic → process exit`, not SIGKILL or eviction:
```
KL Prod:
restartCount: 67 (over ~21h on the fresh disk)
current state: waiting (CrashLoopBackOff)
lastTerminated: reason=Error exitCode=2 (Go runtime panic, NOT 137/OOMKilled)
pod podStartTime: 2026-05-06T09:35:47Z (uninterrupted; same pod object the whole run)
SG Dev:
restartCount: 70 (over ~20h)
lastTerminated: reason=Error exitCode=2
pod podStartTime: 2026-05-06T10:41:07Z (uninterrupted)
```
Both pods are still the original ones from the fresh-disk start — same `metadata.creationTimestamp`, same UID. Just the container has been restarting in place.
`kubectl get events` is no help for the *first* panic (Kubernetes evicts events after ~1h by default and we're 21h in), but the recent windows show only `Unhealthy` (readiness probe vs. a DB that's mid-restart) and `BackOff` — no `OOMKilling`, no `Evicted`, no `Killing`. Pod-level `restartCount` only counts container restarts, and on every previous termination the reason has been `Error` (Go panic), not `OOMKilled`.
### 2. Memory limits + utilization
```
KL Prod banyandb container: requests cpu=2 memory=16Gi limits cpu=8 memory=32Gi
SG Dev banyandb container: requests cpu=1 memory=8Gi limits cpu=4 memory=16Gi
```
Live `kubectl top` snapshot:
```
SG Dev banyandb: 2464m CPU, 4760Mi RSS (≈ 30% of the 16Gi limit, ≈ 60% of the 8Gi request)
KL Prod banyandb: metrics-server returns "Metrics not available, age: 20h36m" — pod has been
restarting too aggressively for metrics-server to settle on a sample.
I'll capture this on the next clean restart and update.
```
We don't have a Prometheus retention long enough to cover the first 30-60 min after the wipe; if it would be useful, I can wipe one cluster again and run `kubectl top --containers` on a 5s loop for the first hour and share the trace.
### 3. An alternative hypothesis for the first tear
Given that exit code 2 is the only termination reason we ever see and there's no OOMKill/eviction signal, I don't think the first torn part comes from an external kill. My guess:
> **The timestamp panic from #13860 fires inside the merge goroutine, gets recovered by an inner gRPC recovery interceptor (not the merger's own goroutine), and leaves the merge mid-flight — torn `spans.bin` with `metadata.json` not yet written.**
The reasoning:
- The very first `cannot contain timestamp smaller than` panic on a fresh KL disk fires within a few minutes of OAP starting to ingest, well before any of the offset panics.
- Cadence on a fresh KL disk: ~7 timestamp panics in the first 31 min, ~31 in the first 2h, all caught by `grpc-middleware/v2` recovery (no process exit). Per-environment frequency tracks ingest volume (KL ≈ 4 min/panic at high traffic, SG ≈ 5 min/panic at dev traffic).
- The first offset panic only shows up after the timestamp panic has fired many times. After that it self-perpetuates exactly as #13862 describes.
If the timestamp panic ever propagates from a write goroutine into a merge-write code path (or shares any I/O state with the merge writer's `seqWriter`s), the torn-write window in the audit is open. This would also explain why the latency to the *first* offset panic correlates loosely with traffic: more timestamp panics → more chances to land one mid-merge.
I haven't proven this — it's a hypothesis from observing the panic ordering. Two ways to discriminate:
a. **Add a panic-counter and timestamp to the merge goroutine.** If timestamp panics are observed inside `mergeParts` / `mustWriteBlock` (i.e., the merge writer's own code path, not the gRPC write handler), this hypothesis is confirmed.
b. **Land #13860's check as a recoverable error in the merge writer too**, not just a `Panicf`. Independent of #13862, that closes a class of "merge goroutine raised a panic from inside an `MustWriteBlock` call and damaged its own output."
### 4. Help we can offer going forward
- I can run an instrumented build on KL Prod whenever you have one ready (e.g., one that logs `(traceID, offsets, bytesRead, file path)` at every `seqWriter` open/close and at every panic site). Iteration time is ~25 min between panics on a fresh disk in KL.
- I can run the standalone reproducer test you described (`Test_merger_tornWriteRepro` in `banyand/trace/merger_repro_test.go`) locally to confirm we reproduce the exact same panic shape — please link the branch when convenient.
- I can take controlled clean wipes on either KL or SG to characterize the first-tear timing as needed.
In the meantime, given that the issue is now well-understood and a fix is in flight, we're going to roll back to BanyanDB 0.9.0 + OAP 10.3.0 to restore monitoring availability. We'll be ready to pick up an 0.10.x release that includes #13862 whenever it's available, and to verify the fix on the same workload that exposed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily closed issue #13861: [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process
URL: https://github.com/apache/skywalking/issues/13861
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4394424605
Thanks for the clean fresh-disk experiment, @Felix-wave — that pinned down "this is not 0.9 file residue."
Update from this side: I have a unit test that reproduces the exact panic message verbatim by truncating `spans.bin` at a block boundary while leaving `metadata.json`/`primary.bin` intact:
```
panic: offset 5300 must be equal to bytesRead 5247
```
Same shape as your `offset 1400877 must be equal to bytesRead 1400490`. The on-disk state required to trigger it is what you'd see after a hard kill (SIGKILL/OOM/eviction) mid-merge: BanyanDB's merge write path doesn't `fsync` data files before writing `metadata.json`, so a kill between writeback and metadata commit leaves a torn part that survives across restarts and re-trips the panic on every wake-up of the merge loop. That matches the ~8-minute crash cadence in your cluster — once you have one torn part, the loop perpetuates.
Audit findings + proposed fix: #13862.
The one piece I haven't been able to pin down from your report is what triggers the *first* tear on a fresh disk — your "first occurrence ~30 min after wipe, then ~25 min cadence" doesn't square with a self-perpetuating cycle alone, since the cycle needs an initial torn part to start. Two quick questions that would discriminate:
1. **Pod exit history before the first panic.** Could you run, on one of the crashing pods:
```
kubectl get events -n <ns> --sort-by='.lastTimestamp' | grep banyandb
kubectl describe pod <banyandb-pod> | grep -A5 "Last State"
```
What I'm looking for: any `OOMKilling`, `Evicted`, `Killing`, or `BackOff` events *before* the first panic-driven exit, and the `Reason` on the previous termination. If we see `OOMKilled` or `Evicted`, that's the first-cause torn write.
2. **Memory limits + utilization.** What's the BanyanDB pod's memory request/limit, and do you have a memory-utilization graph for the first 30-60 minutes of a fresh-disk run? OAP backfill into a fresh BanyanDB hits the merge loop hard right after schema creation; if the pod is brushing the limit there, an OOMKill is the most plausible first trigger.
Independent of what turns out to be the first cause, the durability + read-side fixes proposed in #13862 will stop the perpetuation: once they land, even a torn write from an OOMKill won't take down the merger — the affected part is quarantined and the loop continues.
Happy to share the reproducer test (it's a single self-contained Go test against `tst.mergeParts`) if you'd like to run it locally to convince yourself the panic shape matches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4394035670
Is that test data? could you share the data file(folder) with us to verify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4386489907
According to versions.yml in [v0.10.1](https://github.com/apache/skywalking-banyandb/blob/v0.10.1/banyand/internal/storage/versions.yml), the required file version is 1.4.0. This should be incompatible with version 1.3.0 (used in 0.9.0).
The database is expected to check the file version during bootstrap and stop if it is unsupported. If you managed to start 0.10.1 using 0.9.0 data files, it suggests the version check process is not functioning correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "Felix-wave (via GitHub)" <gi...@apache.org>.
Felix-wave commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4393897787
Thanks for the quick reply, @hanahmily. To verify the version-mismatch hypothesis, we ran a controlled experiment with a fully clean state on both of our environments (KL Prod and SG Dev):
1. Scaled OAP and BanyanDB to 0.
2. **Deleted the PVC entirely**, which (with `reclaimPolicy: Delete` and Aliyun ESSD CSI) cascaded to PV deletion and physical disk deletion. Confirmed both Aliyun disks were gone:
- KL: `d-8ps1hxrabeggd4foci5p` → deleted
- SG: `d-t4niiqzwz9f8wke16x97` → deleted
3. Created a fresh PVC of the same size. Storage class `alicloud-disk-essd` provisioned **brand-new disks** (`d-8ps458950s8goggjqiso` for KL, `d-t4ngw4m8qhuusgrtvnod` for SG).
4. Scaled BanyanDB 0.10.1 + OAP 10.4.0 back up. Verified `/data` was empty on first boot and OAP recreated all groups + schemas from scratch.
So at this point there is **no possibility of 0.9.x file residue** — the disk itself is new, never touched by 0.9.
### Initial observation (looked promising)
For roughly the first 2 hours, the merger panic was absent on both clusters:
```
KL Prod (fresh disk, 2h6m): trace 25.5G / stream 33G #13861 = 0
SG Dev (fresh disk, 1h): trace 166M / stream 5.8G #13861 = 0
```
Notably, KL had already accumulated **25.5 GB of trace data — about 2× the 12.9 GB it had on the old disk when this panic was occurring every ~8 minutes** — yet zero `offset must be equal to bytesRead` panics during this window.
### After more time, the panic returned
I let both clusters run overnight. ~17 hours after starting fresh:
```
KL Prod (fresh disk, 17h):
banyandb restart=41
most recent panic: "offset 0 must be equal to bytesRead 63954"
pod state: CrashLoopBackOff
disk used: 528 GB
SG Dev (fresh disk, 16h):
banyandb restart=70
most recent panic: "offset 5263206 must be equal to bytesRead 5262950"
disk used: 62 GB
```
So the panic does eventually fire on a brand-new disk; it just takes longer than ~2 hours to reach whatever condition triggers it. Restart cadence on the fresh disk in steady state for KL is roughly **once every 25 minutes** (vs ~8 min on the old disk — possibly a function of accumulated state, but still not stable).
### What this rules out
The "0.9.x file residue + missing version check" hypothesis does not explain the failure: the failure reproduces on disks that were never written to by 0.9.x.
### What it suggests
The panic appears tied to **runtime state that 0.10.1 itself produces** (merger output? compaction? specific tag layouts?), not to legacy file format. Happy to dig further — e.g., capture a part dump just before the next panic if there's a tool for it, or attach the full panic context with a few hundred surrounding log lines.
Should I file this as a separate finding, or treat it as a continuation of this issue? Also, is the version-check process itself worth a separate ticket, since you mentioned it might not be functioning correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process [skywalking]
Posted by "Felix-wave (via GitHub)" <gi...@apache.org>.
Felix-wave commented on issue #13861:
URL: https://github.com/apache/skywalking/issues/13861#issuecomment-4394091981
Hi @wu-sheng — this is real production traffic, not test data. Trace payloads contain customer data (full SQL bodies, HTTP request URLs, internal service/endpoint names, infrastructure identifiers), so I'm afraid we can't share the raw `/data` directory or BanyanDB part folders directly.
Happy to collaborate in any other way that helps you reproduce. Some options I can offer, ordered roughly by what should be easiest:
1. **Workload profile, in detail.** I can describe the trace ingestion shape (services, agents, average trace depth, tag distributions, peak trace/sec, segment size distribution) so you can drive a synthetic generator to a comparable load. We're in the millions of segments/day range across ~30 Java services on agent 9.5.0.
2. **Redacted / anonymized dumps.** If there's an existing tool in BanyanDB (or one you can point me to / send me a snippet for) that dumps **only structural metadata of a part** (offsets, sizes, tag-family headers, block boundaries — *no* tag values, no span bytes), I can run it and share the output. The bug seems to be in offset/bytesRead alignment, so structural metadata may already be sufficient to diagnose.
3. **Run an instrumented build for you.** If you publish a debug image/binary with extra logging around `block_writer`/`mergeBlocks`/`partMergeIter.mustReadRaw` — e.g., logging the offsets, bytesRead, traceID, and the blockMetadata being written / read just before each panic — I can run it on KL Prod (high traffic) until we capture a panic and then share the redacted log snippet. We're already seeing this every ~25 min on a fresh disk, so iteration is fast.
4. **Live triage session.** I can pull arbitrary state from the cluster on demand: pod logs, banyandb HTTP API responses, `du` / `find` output of `/data`, schema dumps, OAP startup logs, etc. Tell me what you'd like to see and I'll capture it.
5. **Synthetic reproducer.** If we can pin down what's special in our traffic (e.g., particular tag shape, specific segment patterns), I can build a small load generator that drives the same pattern against a sandbox BanyanDB and share that.
Which of these would be the most useful starting point? Option 3 (an instrumented build) feels highest-signal given the failure mode is clearly inside the trace storage engine's offset bookkeeping, but I'll defer to whatever you think is most efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org