You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "Fine0830 (via GitHub)" <gi...@apache.org> on 2025/12/05 05:05:23 UTC
[PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Fine0830 opened a new pull request, #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886
- [x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Related apache/skywalking#13594
- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md).
## Features
- **Metrics Polling**: Continuously polls Prometheus metrics from BanyanDB container
- **Flight Recorder**: Buffers metrics data in a circular buffer using memory-mapped files, ensuring data survives crashes
- **Sidecar Mode**: Automatic service discovery and coordination when running alongside BanyanDB instances
- **Health Endpoints**: Built-in health endpoints for Kubernetes liveness/readiness probes
Signed-off-by: Qiuxia Fan <qi...@apache.org>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#discussion_r2592868314
##########
docs/operation/fodc.md:
##########
@@ -0,0 +1,294 @@
+# FODC
+
+FODC is a monitoring tool for BanyanDB containers that polls Prometheus metrics and records them in a flight recorder for analysis.
+
+**FODC can run as a Sidecar** alongside each BanyanDB instance, automatically discovering and monitoring the BanyanDB instance in the same pod/container group. This sidecar pattern ensures that diagnostic data collection is coordinated and always available.
+
+## Features
+
+- π **Metrics Polling**: Continuously polls Prometheus metrics from BanyanDB container
+- π― **Flight Recorder**: Buffers metrics data in a circular buffer using memory-mapped files, ensuring data survives crashes
+- π **Sidecar Mode**: Automatic service discovery and coordination when running alongside BanyanDB instances
+- π₯ **Health Endpoints**: Built-in health endpoints for Kubernetes liveness/readiness probes
+
+## Installation
+
+```bash
+cd fodc
+make fodc-cli fodc-view
+```
+
+The binaries will be built at:
+- `build/bin/dev/fodc-cli` - Main monitoring tool
+- `build/bin/dev/fodc-view` - Flight recorder viewer tool
+
+## Usage
+
+### Basic Usage
+
+```bash
+./build/bin/dev/fodc-cli
+```
+
+### With Custom Configuration
+
+```bash
+./build/bin/dev/fodc-cli \
+ --metrics-url=http://localhost:2121/metrics \
+ --poll-interval=5s \
+ --health-url=http://localhost:17913/api/healthz
+```
+
+### Sidecar Mode
+
+FODC can run in **sidecar mode** with automatic service discovery:
+
+```bash
+./build/bin/dev/fodc-cli --sidecar
+```
+
+In sidecar mode, FODC will:
+- Automatically discover BanyanDB endpoints using environment variables or defaults
+- Start a health endpoint server (default port: 17914)
+- Provide Kubernetes-compatible health endpoints (`/healthz`, `/ready`, `/live`)
+- Share the same network namespace as BanyanDB (in Kubernetes/Docker)
+
+**Environment Variables for Sidecar Mode:**
+- `BANYANDB_HOST`: BanyanDB hostname (default: `localhost`)
+- `BANYANDB_METRICS_PORT`: Metrics port (default: `2121`)
+- `BANYANDB_HTTP_PORT`: HTTP API port (default: `17913`)
+- `POD_NAME`: Kubernetes pod name (auto-injected)
+- `POD_NAMESPACE`: Kubernetes pod namespace (auto-injected)
+- `POD_IP`: Kubernetes pod IP (auto-injected)
+
+**Note:** The FODC health endpoint port is controlled by the `--health-port` flag (default: `17914`), not an environment variable.
+
+### Command Line Flags
+
+- `--sidecar`: Run in sidecar mode with auto-discovery (default: `false`)
+- `--metrics-url`: Prometheus metrics endpoint URL (auto-discovered in sidecar mode)
+- `--poll-interval`: Interval for polling metrics (default: `5s`)
+- `--health-url`: Health check endpoint URL (auto-discovered in sidecar mode)
+- `--flight-recorder-path`: Path to flight recorder memory-mapped file (default: `/tmp/fodc-flight-recorder.bin`)
+- `--flight-recorder-buffer`: Number of snapshots to buffer in flight recorder (default: `1000`)
+- `--flight-recorder-rotation`: Interval to automatically clear/rotate flight recorder (default: `0` = disabled). Examples: `24h`, `1h30m`, `30m`
+- `--health-port`: Port for sidecar health endpoint (default: `17914`)
+
+## Sidecar Deployment
+
+### Docker Compose
+
+See `examples/docker-compose-sidecar.yml` for a complete example of FODC running as a sidecar alongside BanyanDB:
+
+```bash
+cd fodc/examples
+docker-compose -f docker-compose-sidecar.yml up
+```
+
+The sidecar will automatically discover BanyanDB on `localhost` (same network namespace) and start monitoring.
+
+### Kubernetes
+
+See `examples/kubernetes-sidecar.yaml` for a complete Kubernetes deployment example:
+
+```bash
+kubectl apply -f examples/kubernetes-sidecar.yaml
+```
+
+The deployment includes:
+- BanyanDB container as the main application
+- FODC sidecar container in the same pod
+- Shared volumes for flight recorder data
+- Health probes for both containers
+- Service discovery via environment variables
+
+**Key Features:**
+- Both containers share the same pod network namespace (use `localhost` to communicate)
+- Shared volumes allow FODC to persist flight recorder data
+- Kubernetes environment variables (`POD_NAME`, `POD_IP`, etc.) are automatically injected
+- Health endpoints enable proper liveness/readiness probe configuration
+
+## Running in Docker
+
+To run FODC inside a Docker container monitoring BanyanDB:
+
+```dockerfile
+FROM golang:1.21-alpine AS builder
+WORKDIR /app
+COPY . .
+RUN go build -o fodc ./cmd/fodc
+
+FROM alpine:latest
+RUN apk --no-cache add ca-certificates
+COPY --from=builder /app/fodc /usr/local/bin/fodc
+ENTRYPOINT ["fodc"]
+```
+
+Or use the provided Dockerfile:
+
+```bash
+cd fodc
+make docker-build
+```
+
+## Integration with BanyanDB
+
+FODC is designed to work with BanyanDB containers. Ensure that:
+
+1. BanyanDB metrics endpoint is exposed (port 2121)
+2. Health check endpoint is accessible (port 17913)
+
+### Sidecar Pattern Benefits
+
+When running as a sidecar, FODC provides several advantages:
+
+1. **Automatic Discovery**: No manual configuration needed - FODC discovers BanyanDB automatically
+2. **Shared Network**: Uses the same network namespace, enabling efficient localhost communication
+3. **Coordinated Lifecycle**: Sidecar starts/stops with the BanyanDB instance
+4. **Shared Storage**: Can share volumes for flight recorder data
+5. **Health Monitoring**: Provides health endpoints for Kubernetes/Docker orchestration
+6. **Per-Instance Monitoring**: Each BanyanDB instance has its own dedicated FODC sidecar
+
+## Flight Recorder
+
+The Flight Recorder is a critical component that ensures metrics data survives crashes:
+
+- **Memory-Mapped Storage**: Uses memory-mapped files for efficient persistence
+- **Circular Buffer**: Implements a circular buffer to store the most recent N snapshots
+- **Crash Recovery**: Automatically recovers data on startup if the process crashed
+- **Independent Memory Space**: Operates in its own memory space, separate from the main process
+
+The flight recorder buffers metrics snapshots in a memory-mapped file. When FODC starts, it automatically attempts to recover any previously recorded data, allowing you to analyze metrics from before a crash.
+
+### Flight Recorder Rotation/Clearing
+
+The flight recorder uses a circular buffer, so old data is automatically overwritten when the buffer is full. However, you can also:
Review Comment:
As the metrics data increases in memory under our control, we should use memory usage as the threshold instead of a duration. Because no matter what, we don't want FODC OOM.
The threshold could be 1. Fixed memory usage(20 MB per FODC node) 2. Usage percentage(20% of container memory). I am not sure whether <2> is possible in go.
@hanahmily @Fine0830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#discussion_r2592856207
##########
docs/operation/fodc.md:
##########
@@ -0,0 +1,294 @@
+# FODC
+
+FODC is a monitoring tool for BanyanDB containers that polls Prometheus metrics and records them in a flight recorder for analysis.
Review Comment:
@hanahmily I am referring to our previous discussion. FODC is not just being as a metric recorder, it is beyond that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#issuecomment-3615360321
## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/886?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 29.06%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fb53aa9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/fb53aa979081219d96985c6f4e14948518d3bcb5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 62 commits behind head on main.
> :exclamation: There is a different number of reports uploaded between BASE (3530dd9) and HEAD (fb53aa9). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (3530dd9) | HEAD (fb53aa9) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #886 +/- ##
===========================================
- Coverage 45.97% 29.06% -16.91%
===========================================
Files 328 114 -214
Lines 55505 11930 -43575
===========================================
- Hits 25520 3468 -22052
+ Misses 27909 8128 -19781
+ Partials 2076 334 -1742
```
| [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/886/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/886/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `29.06% <ΓΈ> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/886?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "Fine0830 (via GitHub)" <gi...@apache.org>.
Fine0830 closed pull request #886: feat: implement the FODC Sidecar
URL: https://github.com/apache/skywalking-banyandb/pull/886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#discussion_r2591657389
##########
fodc/internal/flightrecorder/flightrecorder.go:
##########
@@ -0,0 +1,312 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+// either express or implied. See the License for the specific
+// language governing permissions and limitations under the License.
+
+package flightrecorder
+
+import (
+ "encoding/binary"
+ "encoding/json"
+ "fmt"
+ "os"
+ "sync"
+ "unsafe"
+
+ "github.com/apache/skywalking-banyandb/fodc/internal/poller"
+)
+
+const (
+ // DefaultBufferSize is the default number of snapshots to store
+ DefaultBufferSize = 1000
+ // HeaderSize is the size of the file header in bytes
+ HeaderSize = 64
+ // MagicNumber identifies the flight recorder file format
+ MagicNumber = uint32(0x464C5243) // "FLRC"
+ // Version is the file format version
+ Version = uint32(1)
+)
+
+// Header represents the file header
+type Header struct {
+ Magic uint32 // Magic number
+ Version uint32 // Format version
+ BufferSize uint32 // Number of slots in buffer
+ WriteIndex uint32 // Current write index (circular)
+ Count uint32 // Total number of entries written
+ _ [40]byte // Padding to 64 bytes
+}
+
+// FlightRecorder implements a circular buffer using memory-mapped files
+// to persist metrics data across crashes
+type FlightRecorder struct {
+ file *os.File
+ header *Header
+ data []byte
+ bufferSize uint32
+ slotSize uint32
+ mu sync.RWMutex
+ path string
+}
+
+// NewFlightRecorder creates a new FlightRecorder with the specified buffer size
+func NewFlightRecorder(path string, bufferSize uint32) (*FlightRecorder, error) {
+ if bufferSize == 0 {
+ bufferSize = DefaultBufferSize
+ }
+
+ // Calculate slot size: estimate max snapshot size (1MB per snapshot)
+ slotSize := uint32(1024 * 1024) // 1MB per slot
+ // Total file size: header + (bufferSize * slotSize)
+ fileSize := int64(HeaderSize) + int64(bufferSize)*int64(slotSize)
+
+ // Open or create the file
+ file, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
+ if err != nil {
+ return nil, fmt.Errorf("failed to open flight recorder file: %w", err)
+ }
+
+ // Get file info to check if it exists and has data
+ info, err := file.Stat()
+ if err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to stat file: %w", err)
+ }
+
+ // If file is new or smaller than expected, resize it
+ if info.Size() < fileSize {
+ if err := file.Truncate(fileSize); err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to resize file: %w", err)
+ }
+ }
+
+ // Memory map the file
+ data, err := mmapFile(file, int(fileSize))
+ if err != nil {
+ file.Close()
Review Comment:
File handle may be writable as a result of data flow from a [call to OpenFile](1) and closing it may result in data loss upon failure, which is not handled explicitly.
##########
fodc/internal/flightrecorder/flightrecorder.go:
##########
@@ -0,0 +1,312 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+// either express or implied. See the License for the specific
+// language governing permissions and limitations under the License.
+
+package flightrecorder
+
+import (
+ "encoding/binary"
+ "encoding/json"
+ "fmt"
+ "os"
+ "sync"
+ "unsafe"
+
+ "github.com/apache/skywalking-banyandb/fodc/internal/poller"
+)
+
+const (
+ // DefaultBufferSize is the default number of snapshots to store
+ DefaultBufferSize = 1000
+ // HeaderSize is the size of the file header in bytes
+ HeaderSize = 64
+ // MagicNumber identifies the flight recorder file format
+ MagicNumber = uint32(0x464C5243) // "FLRC"
+ // Version is the file format version
+ Version = uint32(1)
+)
+
+// Header represents the file header
+type Header struct {
+ Magic uint32 // Magic number
+ Version uint32 // Format version
+ BufferSize uint32 // Number of slots in buffer
+ WriteIndex uint32 // Current write index (circular)
+ Count uint32 // Total number of entries written
+ _ [40]byte // Padding to 64 bytes
+}
+
+// FlightRecorder implements a circular buffer using memory-mapped files
+// to persist metrics data across crashes
+type FlightRecorder struct {
+ file *os.File
+ header *Header
+ data []byte
+ bufferSize uint32
+ slotSize uint32
+ mu sync.RWMutex
+ path string
+}
+
+// NewFlightRecorder creates a new FlightRecorder with the specified buffer size
+func NewFlightRecorder(path string, bufferSize uint32) (*FlightRecorder, error) {
+ if bufferSize == 0 {
+ bufferSize = DefaultBufferSize
+ }
+
+ // Calculate slot size: estimate max snapshot size (1MB per snapshot)
+ slotSize := uint32(1024 * 1024) // 1MB per slot
+ // Total file size: header + (bufferSize * slotSize)
+ fileSize := int64(HeaderSize) + int64(bufferSize)*int64(slotSize)
+
+ // Open or create the file
+ file, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
+ if err != nil {
+ return nil, fmt.Errorf("failed to open flight recorder file: %w", err)
+ }
+
+ // Get file info to check if it exists and has data
+ info, err := file.Stat()
+ if err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to stat file: %w", err)
+ }
+
+ // If file is new or smaller than expected, resize it
+ if info.Size() < fileSize {
+ if err := file.Truncate(fileSize); err != nil {
+ file.Close()
Review Comment:
File handle may be writable as a result of data flow from a [call to OpenFile](1) and closing it may result in data loss upon failure, which is not handled explicitly.
```suggestion
closeErr := file.Close()
if closeErr != nil {
return nil, fmt.Errorf("failed to resize file: %v; additionally, failed to close file: %v", err, closeErr)
}
```
##########
fodc/internal/flightrecorder/flightrecorder.go:
##########
@@ -0,0 +1,312 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+// either express or implied. See the License for the specific
+// language governing permissions and limitations under the License.
+
+package flightrecorder
+
+import (
+ "encoding/binary"
+ "encoding/json"
+ "fmt"
+ "os"
+ "sync"
+ "unsafe"
+
+ "github.com/apache/skywalking-banyandb/fodc/internal/poller"
+)
+
+const (
+ // DefaultBufferSize is the default number of snapshots to store
+ DefaultBufferSize = 1000
+ // HeaderSize is the size of the file header in bytes
+ HeaderSize = 64
+ // MagicNumber identifies the flight recorder file format
+ MagicNumber = uint32(0x464C5243) // "FLRC"
+ // Version is the file format version
+ Version = uint32(1)
+)
+
+// Header represents the file header
+type Header struct {
+ Magic uint32 // Magic number
+ Version uint32 // Format version
+ BufferSize uint32 // Number of slots in buffer
+ WriteIndex uint32 // Current write index (circular)
+ Count uint32 // Total number of entries written
+ _ [40]byte // Padding to 64 bytes
+}
+
+// FlightRecorder implements a circular buffer using memory-mapped files
+// to persist metrics data across crashes
+type FlightRecorder struct {
+ file *os.File
+ header *Header
+ data []byte
+ bufferSize uint32
+ slotSize uint32
+ mu sync.RWMutex
+ path string
+}
+
+// NewFlightRecorder creates a new FlightRecorder with the specified buffer size
+func NewFlightRecorder(path string, bufferSize uint32) (*FlightRecorder, error) {
+ if bufferSize == 0 {
+ bufferSize = DefaultBufferSize
+ }
+
+ // Calculate slot size: estimate max snapshot size (1MB per snapshot)
+ slotSize := uint32(1024 * 1024) // 1MB per slot
+ // Total file size: header + (bufferSize * slotSize)
+ fileSize := int64(HeaderSize) + int64(bufferSize)*int64(slotSize)
+
+ // Open or create the file
+ file, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
+ if err != nil {
+ return nil, fmt.Errorf("failed to open flight recorder file: %w", err)
+ }
+
+ // Get file info to check if it exists and has data
+ info, err := file.Stat()
+ if err != nil {
+ file.Close()
Review Comment:
File handle may be writable as a result of data flow from a [call to OpenFile](1) and closing it may result in data loss upon failure, which is not handled explicitly.
##########
fodc/internal/flightrecorder/flightrecorder.go:
##########
@@ -0,0 +1,312 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+// either express or implied. See the License for the specific
+// language governing permissions and limitations under the License.
+
+package flightrecorder
+
+import (
+ "encoding/binary"
+ "encoding/json"
+ "fmt"
+ "os"
+ "sync"
+ "unsafe"
+
+ "github.com/apache/skywalking-banyandb/fodc/internal/poller"
+)
+
+const (
+ // DefaultBufferSize is the default number of snapshots to store
+ DefaultBufferSize = 1000
+ // HeaderSize is the size of the file header in bytes
+ HeaderSize = 64
+ // MagicNumber identifies the flight recorder file format
+ MagicNumber = uint32(0x464C5243) // "FLRC"
+ // Version is the file format version
+ Version = uint32(1)
+)
+
+// Header represents the file header
+type Header struct {
+ Magic uint32 // Magic number
+ Version uint32 // Format version
+ BufferSize uint32 // Number of slots in buffer
+ WriteIndex uint32 // Current write index (circular)
+ Count uint32 // Total number of entries written
+ _ [40]byte // Padding to 64 bytes
+}
+
+// FlightRecorder implements a circular buffer using memory-mapped files
+// to persist metrics data across crashes
+type FlightRecorder struct {
+ file *os.File
+ header *Header
+ data []byte
+ bufferSize uint32
+ slotSize uint32
+ mu sync.RWMutex
+ path string
+}
+
+// NewFlightRecorder creates a new FlightRecorder with the specified buffer size
+func NewFlightRecorder(path string, bufferSize uint32) (*FlightRecorder, error) {
+ if bufferSize == 0 {
+ bufferSize = DefaultBufferSize
+ }
+
+ // Calculate slot size: estimate max snapshot size (1MB per snapshot)
+ slotSize := uint32(1024 * 1024) // 1MB per slot
+ // Total file size: header + (bufferSize * slotSize)
+ fileSize := int64(HeaderSize) + int64(bufferSize)*int64(slotSize)
+
+ // Open or create the file
+ file, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
+ if err != nil {
+ return nil, fmt.Errorf("failed to open flight recorder file: %w", err)
+ }
+
+ // Get file info to check if it exists and has data
+ info, err := file.Stat()
+ if err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to stat file: %w", err)
+ }
+
+ // If file is new or smaller than expected, resize it
+ if info.Size() < fileSize {
+ if err := file.Truncate(fileSize); err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to resize file: %w", err)
+ }
+ }
+
+ // Memory map the file
+ data, err := mmapFile(file, int(fileSize))
+ if err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to memory map file: %w", err)
+ }
+
+ // Read or initialize header
+ header := (*Header)(unsafe.Pointer(&data[0]))
+
+ if header.Magic != MagicNumber {
+ // Initialize new header
+ header.Magic = MagicNumber
+ header.Version = Version
+ header.BufferSize = bufferSize
+ header.WriteIndex = 0
+ header.Count = 0
+ // Sync header immediately
+ if err := msync(data[:HeaderSize]); err != nil {
+ munmap(data)
+ file.Close()
+ return nil, fmt.Errorf("failed to sync header: %w", err)
+ }
+ } else if header.Version != Version {
+ munmap(data)
+ file.Close()
Review Comment:
File handle may be writable as a result of data flow from a [call to OpenFile](1) and closing it may result in data loss upon failure, which is not handled explicitly.
##########
fodc/internal/flightrecorder/flightrecorder.go:
##########
@@ -0,0 +1,312 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+// either express or implied. See the License for the specific
+// language governing permissions and limitations under the License.
+
+package flightrecorder
+
+import (
+ "encoding/binary"
+ "encoding/json"
+ "fmt"
+ "os"
+ "sync"
+ "unsafe"
+
+ "github.com/apache/skywalking-banyandb/fodc/internal/poller"
+)
+
+const (
+ // DefaultBufferSize is the default number of snapshots to store
+ DefaultBufferSize = 1000
+ // HeaderSize is the size of the file header in bytes
+ HeaderSize = 64
+ // MagicNumber identifies the flight recorder file format
+ MagicNumber = uint32(0x464C5243) // "FLRC"
+ // Version is the file format version
+ Version = uint32(1)
+)
+
+// Header represents the file header
+type Header struct {
+ Magic uint32 // Magic number
+ Version uint32 // Format version
+ BufferSize uint32 // Number of slots in buffer
+ WriteIndex uint32 // Current write index (circular)
+ Count uint32 // Total number of entries written
+ _ [40]byte // Padding to 64 bytes
+}
+
+// FlightRecorder implements a circular buffer using memory-mapped files
+// to persist metrics data across crashes
+type FlightRecorder struct {
+ file *os.File
+ header *Header
+ data []byte
+ bufferSize uint32
+ slotSize uint32
+ mu sync.RWMutex
+ path string
+}
+
+// NewFlightRecorder creates a new FlightRecorder with the specified buffer size
+func NewFlightRecorder(path string, bufferSize uint32) (*FlightRecorder, error) {
+ if bufferSize == 0 {
+ bufferSize = DefaultBufferSize
+ }
+
+ // Calculate slot size: estimate max snapshot size (1MB per snapshot)
+ slotSize := uint32(1024 * 1024) // 1MB per slot
+ // Total file size: header + (bufferSize * slotSize)
+ fileSize := int64(HeaderSize) + int64(bufferSize)*int64(slotSize)
+
+ // Open or create the file
+ file, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
+ if err != nil {
+ return nil, fmt.Errorf("failed to open flight recorder file: %w", err)
+ }
+
+ // Get file info to check if it exists and has data
+ info, err := file.Stat()
+ if err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to stat file: %w", err)
+ }
+
+ // If file is new or smaller than expected, resize it
+ if info.Size() < fileSize {
+ if err := file.Truncate(fileSize); err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to resize file: %w", err)
+ }
+ }
+
+ // Memory map the file
+ data, err := mmapFile(file, int(fileSize))
+ if err != nil {
+ file.Close()
+ return nil, fmt.Errorf("failed to memory map file: %w", err)
+ }
+
+ // Read or initialize header
+ header := (*Header)(unsafe.Pointer(&data[0]))
+
+ if header.Magic != MagicNumber {
+ // Initialize new header
+ header.Magic = MagicNumber
+ header.Version = Version
+ header.BufferSize = bufferSize
+ header.WriteIndex = 0
+ header.Count = 0
+ // Sync header immediately
+ if err := msync(data[:HeaderSize]); err != nil {
+ munmap(data)
+ file.Close()
Review Comment:
File handle may be writable as a result of data flow from a [call to OpenFile](1) and closing it may result in data loss upon failure, which is not handled explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#discussion_r2592850716
##########
docs/operation/fodc.md:
##########
@@ -0,0 +1,294 @@
+# FODC
+
+FODC is a monitoring tool for BanyanDB containers that polls Prometheus metrics and records them in a flight recorder for analysis.
Review Comment:
```suggestion
FODC is a monitoring tool for BanyanDB containers that uses various toolings to measure and profile database node performance, including
1. Scrape Prometheus metrics and record them in a flight recorder for analysis.
2. Use profiling(e.g., eBPF) tools to measure performance from OS perspective.
3. Snapshot database node memory and runtime(e.g., goroutine) status for further analysis.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#discussion_r2592868314
##########
docs/operation/fodc.md:
##########
@@ -0,0 +1,294 @@
+# FODC
+
+FODC is a monitoring tool for BanyanDB containers that polls Prometheus metrics and records them in a flight recorder for analysis.
+
+**FODC can run as a Sidecar** alongside each BanyanDB instance, automatically discovering and monitoring the BanyanDB instance in the same pod/container group. This sidecar pattern ensures that diagnostic data collection is coordinated and always available.
+
+## Features
+
+- π **Metrics Polling**: Continuously polls Prometheus metrics from BanyanDB container
+- π― **Flight Recorder**: Buffers metrics data in a circular buffer using memory-mapped files, ensuring data survives crashes
+- π **Sidecar Mode**: Automatic service discovery and coordination when running alongside BanyanDB instances
+- π₯ **Health Endpoints**: Built-in health endpoints for Kubernetes liveness/readiness probes
+
+## Installation
+
+```bash
+cd fodc
+make fodc-cli fodc-view
+```
+
+The binaries will be built at:
+- `build/bin/dev/fodc-cli` - Main monitoring tool
+- `build/bin/dev/fodc-view` - Flight recorder viewer tool
+
+## Usage
+
+### Basic Usage
+
+```bash
+./build/bin/dev/fodc-cli
+```
+
+### With Custom Configuration
+
+```bash
+./build/bin/dev/fodc-cli \
+ --metrics-url=http://localhost:2121/metrics \
+ --poll-interval=5s \
+ --health-url=http://localhost:17913/api/healthz
+```
+
+### Sidecar Mode
+
+FODC can run in **sidecar mode** with automatic service discovery:
+
+```bash
+./build/bin/dev/fodc-cli --sidecar
+```
+
+In sidecar mode, FODC will:
+- Automatically discover BanyanDB endpoints using environment variables or defaults
+- Start a health endpoint server (default port: 17914)
+- Provide Kubernetes-compatible health endpoints (`/healthz`, `/ready`, `/live`)
+- Share the same network namespace as BanyanDB (in Kubernetes/Docker)
+
+**Environment Variables for Sidecar Mode:**
+- `BANYANDB_HOST`: BanyanDB hostname (default: `localhost`)
+- `BANYANDB_METRICS_PORT`: Metrics port (default: `2121`)
+- `BANYANDB_HTTP_PORT`: HTTP API port (default: `17913`)
+- `POD_NAME`: Kubernetes pod name (auto-injected)
+- `POD_NAMESPACE`: Kubernetes pod namespace (auto-injected)
+- `POD_IP`: Kubernetes pod IP (auto-injected)
+
+**Note:** The FODC health endpoint port is controlled by the `--health-port` flag (default: `17914`), not an environment variable.
+
+### Command Line Flags
+
+- `--sidecar`: Run in sidecar mode with auto-discovery (default: `false`)
+- `--metrics-url`: Prometheus metrics endpoint URL (auto-discovered in sidecar mode)
+- `--poll-interval`: Interval for polling metrics (default: `5s`)
+- `--health-url`: Health check endpoint URL (auto-discovered in sidecar mode)
+- `--flight-recorder-path`: Path to flight recorder memory-mapped file (default: `/tmp/fodc-flight-recorder.bin`)
+- `--flight-recorder-buffer`: Number of snapshots to buffer in flight recorder (default: `1000`)
+- `--flight-recorder-rotation`: Interval to automatically clear/rotate flight recorder (default: `0` = disabled). Examples: `24h`, `1h30m`, `30m`
+- `--health-port`: Port for sidecar health endpoint (default: `17914`)
+
+## Sidecar Deployment
+
+### Docker Compose
+
+See `examples/docker-compose-sidecar.yml` for a complete example of FODC running as a sidecar alongside BanyanDB:
+
+```bash
+cd fodc/examples
+docker-compose -f docker-compose-sidecar.yml up
+```
+
+The sidecar will automatically discover BanyanDB on `localhost` (same network namespace) and start monitoring.
+
+### Kubernetes
+
+See `examples/kubernetes-sidecar.yaml` for a complete Kubernetes deployment example:
+
+```bash
+kubectl apply -f examples/kubernetes-sidecar.yaml
+```
+
+The deployment includes:
+- BanyanDB container as the main application
+- FODC sidecar container in the same pod
+- Shared volumes for flight recorder data
+- Health probes for both containers
+- Service discovery via environment variables
+
+**Key Features:**
+- Both containers share the same pod network namespace (use `localhost` to communicate)
+- Shared volumes allow FODC to persist flight recorder data
+- Kubernetes environment variables (`POD_NAME`, `POD_IP`, etc.) are automatically injected
+- Health endpoints enable proper liveness/readiness probe configuration
+
+## Running in Docker
+
+To run FODC inside a Docker container monitoring BanyanDB:
+
+```dockerfile
+FROM golang:1.21-alpine AS builder
+WORKDIR /app
+COPY . .
+RUN go build -o fodc ./cmd/fodc
+
+FROM alpine:latest
+RUN apk --no-cache add ca-certificates
+COPY --from=builder /app/fodc /usr/local/bin/fodc
+ENTRYPOINT ["fodc"]
+```
+
+Or use the provided Dockerfile:
+
+```bash
+cd fodc
+make docker-build
+```
+
+## Integration with BanyanDB
+
+FODC is designed to work with BanyanDB containers. Ensure that:
+
+1. BanyanDB metrics endpoint is exposed (port 2121)
+2. Health check endpoint is accessible (port 17913)
+
+### Sidecar Pattern Benefits
+
+When running as a sidecar, FODC provides several advantages:
+
+1. **Automatic Discovery**: No manual configuration needed - FODC discovers BanyanDB automatically
+2. **Shared Network**: Uses the same network namespace, enabling efficient localhost communication
+3. **Coordinated Lifecycle**: Sidecar starts/stops with the BanyanDB instance
+4. **Shared Storage**: Can share volumes for flight recorder data
+5. **Health Monitoring**: Provides health endpoints for Kubernetes/Docker orchestration
+6. **Per-Instance Monitoring**: Each BanyanDB instance has its own dedicated FODC sidecar
+
+## Flight Recorder
+
+The Flight Recorder is a critical component that ensures metrics data survives crashes:
+
+- **Memory-Mapped Storage**: Uses memory-mapped files for efficient persistence
+- **Circular Buffer**: Implements a circular buffer to store the most recent N snapshots
+- **Crash Recovery**: Automatically recovers data on startup if the process crashed
+- **Independent Memory Space**: Operates in its own memory space, separate from the main process
+
+The flight recorder buffers metrics snapshots in a memory-mapped file. When FODC starts, it automatically attempts to recover any previously recorded data, allowing you to analyze metrics from before a crash.
+
+### Flight Recorder Rotation/Clearing
+
+The flight recorder uses a circular buffer, so old data is automatically overwritten when the buffer is full. However, you can also:
Review Comment:
As the metrics data increases in memory under our control, we should use memory usage as the threshold instead of a duration. Because no matter what, we don't want FODC OOM.
The threshold could be 1. Fixed memory usage(20 MB per FODC node) 2. Usage percentage(20% of container memory). I am not sure whether <2> is possible in go.
@hanahmily @Fine0830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: implement the FODC Sidecar [skywalking-banyandb]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #886:
URL: https://github.com/apache/skywalking-banyandb/pull/886#discussion_r2592837181
##########
docs/operation/fodc.md:
##########
@@ -0,0 +1,294 @@
+# FODC
Review Comment:
```suggestion
# First Occurrence Data Collection(FODC)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org